Details
-
Bug
-
Resolution: Duplicate
-
Critical
-
Cheshire-Cat
-
Enterprise Edition 7.0.0 build 2241
-
Untriaged
-
Centos 64-bit
-
1
-
Unknown
Description
Build: 7.0.0-2241
Steps to reproduce:
- Cluster
+----------------+----------+-----------------+------------+------------+-----------------------+-------------------+-----------------------+
| Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version |
+----------------+----------+-----------------+------------+------------+-----------------------+-------------------+-----------------------+
| 10.112.191.103 | kv | 60 | 1930829824 | 1290747904 | 1576960 / 1107292160 | 300 / 300 | 7.0.0-2241-enterprise |
| 10.112.191.104 | kv | 56.976744186 | 1042685952 | 566460416 | 11079680 / 1107292160 | 300 / 300 | 7.0.0-2241-enterprise |
+----------------+----------+-----------------+------------+------------+-----------------------+-------------------+-----------------------+
-
+---------------------+---------+----------+-----+-------+-----------+----------+-----------+
| Bucket | Type | Replicas | TTL | Items | RAM Quota | RAM Used | Disk Used |
+---------------------+---------+----------+-----+-------+-----------+----------+-----------+
| hzBq1Pn80gc-2Q9QTs5 | membase | 1 | 0 | 600 | 536870912 | 51784400 | 37181934 |
+---------------------+---------+----------+-----+-------+-----------+----------+-----------+
- Create 2 custom scope and three collections under each scope and load 100 docs into each collection (all are async docs)
- Load docs with key 'test_doc-000' under two different collections at the same time (With doc_durability level=MAJORITY)
- Delete all scope and collection from the bucket
Observation:
Cbstats scope or collection does not contain any items.
]# /opt/couchbase/bin/cbstats 10.112.191.103:11210 -u Administrator -p password collections -b hzBq1Pn80gc-2Q9QTs5
|
manifest_uid: 15
|
]# /opt/couchbase/bin/cbstats 10.112.191.103:11210 -u Administrator -p password scopes -b hzBq1Pn80gc-2Q9QTs5
|
0x0:collections: 0
|
0x0:items: 0
|
0x0:mem_used: 0
|
0x0:name: _default
|
0x0:ops_delete: 0
|
0x0:ops_get: 0
|
0x0:ops_store: 0
|
manifest_uid: 15
|
|
]# /opt/couchbase/bin/cbstats 10.112.191.104:11210 -u Administrator -p password collections -b hzBq1Pn80gc-2Q9QTs5
|
manifest_uid: 15
|
]# /opt/couchbase/bin/cbstats 10.112.191.104:11210 -u Administrator -p password scopes -b hzBq1Pn80gc-2Q9QTs5
|
0x0:collections: 0
|
0x0:items: 0
|
0x0:mem_used: 0
|
0x0:name: _default
|
0x0:ops_delete: 0
|
0x0:ops_get: 0
|
0x0:ops_store: 0
|
manifest_uid: 15
|
But REST / UI reports that there are two docs in the bucket.
]# curl -X GET -u Administrator:password 10.112.191.103:8091/pools/default/buckets/hzBq1Pn80gc-2Q9QTs5
|
{"name":"hzBq1Pn80gc-2Q9QTs5","uuid":"1f715af377d67411d60c86c29b970d24","bucketType":"membase","authType":"sasl","uri":"/pools/default/buckets/hzBq1Pn80gc-2Q9QTs5?bucket_uuid=1f715af377d67411d60c86c29b970d24","streamingUri":"/pools/default/bucketsStreaming/hzBq1Pn80gc-2Q9QTs5?bucket_uuid=1f715af377d67411d60c86c29b970d24","localRandomKeyUri":"/pools/default/buckets/hzBq1Pn80gc-2Q9QTs5/localRandomKey","controllers":{"compactAll":"/pools/default/buckets/hzBq1Pn80gc-2Q9QTs5/controller/compactBucket","compactDB":"/pools/default/buckets/hzBq1Pn80gc-2Q9QTs5/controller/compactDatabases","purgeDeletes":"/pools/default/buckets/hzBq1Pn80gc-2Q9QTs5/controller/unsafePurgeBucket","startRecovery":"/pools/default/buckets/hzBq1Pn80gc-2Q9QTs5/controller/startRecovery"},"nodes":[{"couchApiBaseHTTPS":"https://10.112.191.104:18092/hzBq1Pn80gc-2Q9QTs5%2B1f715af377d67411d60c86c29b970d24","couchApiBase":"http://10.112.191.104:8092/hzBq1Pn80gc-2Q9QTs5%2B1f715af377d67411d60c86c29b970d24","systemStats":{"cpu_utilization_rate":6.25,"cpu_stolen_rate":0,"swap_total":1107292160,"swap_used":20402176,"mem_total":1042685952,"mem_free":576970752,"mem_limit":1042685952,"cpu_cores_available":1,"allocstall":815},"interestingStats":{"cmd_get":0,"couch_docs_actual_disk_size":18116374,"couch_docs_data_size":8518656,"couch_spatial_data_size":0,"couch_spatial_disk_size":0,"couch_views_actual_disk_size":0,"couch_views_data_size":0,"curr_items":0,"curr_items_tot":2,"ep_bg_fetched":0,"get_hits":0,"mem_used":24694168,"ops":0,"vb_active_num_non_resident":0,"vb_replica_curr_items":2},"uptime":"18041","memoryTotal":1042685952,"memoryFree":576970752,"mcdMemoryReserved":795,"mcdMemoryAllocated":795,"replication":1,"clusterMembership":"active","recoveryType":"none","status":"healthy","otpNode":"ns_1@10.112.191.104","hostname":"10.112.191.104:8091","nodeUUID":"d348b48f75232412f6a879ef95080d6b","clusterCompatibility":458752,"version":"7.0.0-2241-enterprise","os":"x86_64-unknown-linux-gnu","cpuCount":1,"ports":{"direct":11210,"httpsCAPI":18092,"httpsMgmt":18091,"distTCP":21100,"distTLS":21150},"services":["kv"],"nodeEncryption":false,"configuredHostname":"10.112.191.104:8091","addressFamily":"inet","externalListeners":[{"afamily":"inet","nodeEncryption":false},{"afamily":"inet6","nodeEncryption":false}]},{"couchApiBaseHTTPS":"https://10.112.191.103:18092/hzBq1Pn80gc-2Q9QTs5%2B1f715af377d67411d60c86c29b970d24","couchApiBase":"http://10.112.191.103:8092/hzBq1Pn80gc-2Q9QTs5%2B1f715af377d67411d60c86c29b970d24","systemStats":{"cpu_utilization_rate":8.695652173913043,"cpu_stolen_rate":0,"swap_total":1107292160,"swap_used":2846720,"mem_total":1930829824,"mem_free":1316626432,"mem_limit":1930829824,"cpu_cores_available":1,"allocstall":1071},"interestingStats":{"cmd_get":0,"couch_docs_actual_disk_size":18121650,"couch_docs_data_size":8530944,"couch_spatial_data_size":0,"couch_spatial_disk_size":0,"couch_views_actual_disk_size":0,"couch_views_data_size":0,"curr_items":2,"curr_items_tot":2,"ep_bg_fetched":0,"get_hits":0,"mem_used":24692584,"ops":0,"vb_active_num_non_resident":0,"vb_replica_curr_items":0},"uptime":"18094","memoryTotal":1930829824,"memoryFree":1316626432,"mcdMemoryReserved":1473,"mcdMemoryAllocated":1473,"replication":1,"clusterMembership":"active","recoveryType":"none","status":"healthy","otpNode":"ns_1@10.112.191.103","thisNode":true,"hostname":"10.112.191.103:8091","nodeUUID":"f7ccd7c43c10085d5c6e6c2d1a777ddb","clusterCompatibility":458752,"version":"7.0.0-2241-enterprise","os":"x86_64-unknown-linux-gnu","cpuCount":1,"ports":{"direct":11210,"httpsCAPI":18092,"httpsMgmt":18091,"distTCP":21100,"distTLS":21150},"services":["kv"],"nodeEncryption":false,"configuredHostname":"10.112.191.103:8091","addressFamily":"inet","externalListeners":[{"afamily":"inet","nodeEncryption":false},{"afamily":"inet6","nodeEncryption":false}]}],"stats":{"uri":"/pools/default/buckets/hzBq1Pn80gc-2Q9QTs5/stats","directoryURI":"/pools/default/buckets/hzBq1Pn80gc-2Q9QTs5/statsDirectory","nodeStatsListURI":"/pools/default/buckets/hzBq1Pn80gc-2Q9QTs5/nodes"},"nodeLocator":"vbucket","saslPassword":"a3434f2d5afa080973f90b5f5149b340","ddocs":{"uri":"/pools/default/buckets/hzBq1Pn80gc-2Q9QTs5/ddocs"},"replicaIndex":true,"autoCompactionSettings":false,"vBucketServerMap":{"hashAlgorithm":"CRC","numReplicas":1,"serverList":["10.112.191.103:11210","10.112.191.104:11210"],"vBucketMap":[[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0]]},"maxTTL":0,"compressionMode":"passive","replicaNumber":1,"threadsNumber":3,"quota":{"ram":536870912,"rawRAM":268435456},"basicStats":{"quotaPercentUsed":9.198999404907227,"opsPerSec":0,"diskFetches":0,"itemCount":2,"diskUsed":36238024,"dataUsed":17049600,"memUsed":49386752,"vbActiveNumNonResident":0},"evictionPolicy":"valueOnly","storageBackend":"couchstore","durabilityMinLevel":"none","fragmentationPercentage":50,"conflictResolutionType":"seqno","bucketCapabilitiesVer":"","bucketCapabilities":["collections","durableWrite","couchapi","dcp","cbhello","touch","cccp","xdcrCheckpointing","nodesExt","xattr"]}
|
Cb_collect logs:
https://cb-jira.s3.us-east-2.amazonaws.com/logs/stale_docs/collectinfo-2020-06-04T200518-ns_1%4010.112.191.103.zip
https://cb-jira.s3.us-east-2.amazonaws.com/logs/stale_docs/collectinfo-2020-06-04T200518-ns_1%4010.112.191.104.zip
Attaching test.log and pcap for further debugging.