Details
-
Bug
-
Resolution: Duplicate
-
Major
-
None
-
4.1.0
-
None
-
Untriaged
-
Ubuntu 64-bit
-
Unknown
Description
Heavy CPU load during each fetch of expired but not removed from disk docs is possible.
Timeframe of high CPU load depends on number of expired, but not removed docs (for example, for 3 nodes x 4CPU and 1M expired documents problem makes [50..70]% util for all cpus on all nodes for ~1 min).
How to reproduce:
1. Establish 3-node cluster (4.1.0-EE-5005, 4CPU, 4GB RAM; 4.1.2-6036 is also affected), all services are enabled
2. Create default bucket (2400M, 1 replica, view index replica, no auth, flush=enable, default compation settings, i/o priority=high)
3. Run following code (or write your own, it just inserts 1M empty docs to default with expiry=current_time+600s ):
package expireddocserrors;
|
|
import com.couchbase.client.java.Bucket;
|
import com.couchbase.client.java.Cluster;
|
import com.couchbase.client.java.CouchbaseCluster;
|
import com.couchbase.client.java.document.JsonDocument;
|
import com.couchbase.client.java.document.json.JsonObject;
|
import com.couchbase.client.java.env.CouchbaseEnvironment;
|
import com.couchbase.client.java.env.DefaultCouchbaseEnvironment;
|
import java.util.LinkedList;
|
import java.util.concurrent.Phaser;
|
|
public class ExpiredDocsErrors extends Thread {
|
|
private final int tid;
|
|
// configure here
|
private static final int threads = 20;
|
private static final int docsPerThread = 50000;
|
private static final int docTTLms = 600 * 1000;
|
private static final int dumpToConsoleEachNDocs = 1000;
|
|
private static final Phaser phaser = new Phaser(threads + 1);
|
private static final CouchbaseEnvironment ce;
|
private static final Cluster cluster;
|
private static final String bucket = "default";
|
|
static {
|
ce = DefaultCouchbaseEnvironment.create();
|
final LinkedList<String> nodes = new LinkedList();
|
nodes.add("A.node");
|
nodes.add("B.node");
|
nodes.add("C.node");
|
cluster = CouchbaseCluster.create(ce, nodes);
|
final Bucket b = cluster.openBucket(bucket);
|
}
|
public ExpiredDocsErrors(final int tid) {
|
this.tid = tid;
|
}
|
public final void run() {
|
try {
|
Bucket b = null;
|
synchronized(cluster) { b = cluster.openBucket(bucket); }
|
final long stm = System.currentTimeMillis();
|
final JsonObject jo = JsonObject.empty();
|
for(int i = 0; i< docsPerThread; i++) {
|
b.upsert(JsonDocument.create(
|
tid + ":" + System.currentTimeMillis(),
|
(int)((System.currentTimeMillis() + docTTLms) / 1000),
|
jo)
|
);
|
if (i % dumpToConsoleEachNDocs == 0) System.out.println("T[" + tid + "] = " + i);
|
}
|
} catch(final Exception e) {
|
e.printStackTrace();
|
} finally {
|
phaser.arriveAndAwaitAdvance();
|
}
|
}
|
public static void main(String[] args) {
|
for(int i = 0; i< threads; i++) new ExpiredDocsErrors(i).start();
|
phaser.arriveAndAwaitAdvance();
|
System.out.println("DONE");
|
}
|
|
}
|
|
4. Wait 10min after done (all docs should expire)
5. Open UI, click on "data buckets" and press "Documents" for "default" bucket
6. You can repeat this as many times, as you wish, CPU spikes anyway
7. There are also babysitter log messages per each "Documents" button press:
memcached<0.75.0>: 2016-05-27T13:12:12.256765Z WARNING (default) Warning: failed to fetch data from database, vBucket=256 key=16:1464353480734 error=document not found [none]
|
memcached<0.75.0>: 2016-05-27T13:12:12.256998Z WARNING (default) Warning: failed to fetch data from database, vBucket=9 key=16:1464353480737 error=document not found [none]
|
memcached<0.75.0>: 2016-05-27T13:12:12.257022Z WARNING (default) Warning: failed to fetch data from database, vBucket=262 key=16:1464353480771 error=document not found [none]
|
memcached<0.75.0>: 2016-05-27T13:12:12.257243Z WARNING (default) Warning: failed to fetch data from database, vBucket=69 key=16:1464353480789 error=document not found [none]
|
According to "sherlok"'s source, there is only one place, where this could happen:
2150@couchstore/src/couch-kvstore.cc
if (errCode != COUCHSTORE_SUCCESS && !meta_only) {
|
LOG(EXTENSION_LOG_WARNING, "Warning: failed to fetch data from database, "
|
"vBucket=%d key=%s error=%s [%s]", cbCtx->vbId,
|
keyStr.c_str(), couchstore_strerror(errCode),
|
couchkvstore_strerrno(db, errCode).c_str());
|
st.numGetFailure++;
|
}
|
Related forum thread: https://forums.couchbase.com/t/4-1-x-all-nodes-are-100-load-strange-memcached-log/8442
Attachments
Issue Links
- duplicates
-
MB-16663 Documents request on UI console generate high ops/sec for nearly a minute and then -> Error: (unexpected)
- Closed