Details
-
Bug
-
Resolution: Not a Bug
-
Major
-
None
-
2.3.0
-
1
Description
This is a weird scenario and I think this is more for SGW team than us, but first a confirmation from Simon Murray .
SGW: 2.8.3-enterprise
CB: 7.0.3
Steps to Reproduce:
- Bring up DAC, Operator pod using latest package.
- Create a 3 node CB cluster defined as:
---
apiVersion: couchbase.com/v2
kind: CouchbaseCluster
metadata:
name: cb-example
spec:
image: couchbase/server:7.0.3
security:
adminSecret: cb-example-auth
buckets:
managed: true
servers:
- size: 3
name: all_services
services:
- data
- index
- query
- search
- eventing
- analytics
volumeMounts:
default: couchbase
pod:
spec:
imagePullSecrets:
- name: test
volumeClaimTemplates:
- metadata:
name: couchbase
spec:
storageClassName: gp2
resources:
requests:
storage: 1Gi
monitoring:
prometheus:
enabled: true
image: registry.gitlab.com/cb-vanilla/exporter:1.0.6-108
logging:
server:
enabled: true
manageConfiguration: true
configurationName: "fluent-bit-config"
sidecar:
image: registry.gitlab.com/cb-vanilla/fluent-bit:1.2.0-101
configurationMountPath: "/fluent-bit/config/"
resources:
requests:
cpu: 100m
memory: 500Mi
- Load 10k items through pillowfight yaml present in the package.
- Create 2 SGW pods running 2.8.3-enterprise image.
- Status of the pods:
Prateeks-MacBook-Pro:k8spkgval prateekkumar$ kubectl get pods
NAME READY STATUS RESTARTS AGE
cb-example-0000 3/3 Running 0 5m12s
cb-example-0001 3/3 Running 0 4m38s
cb-example-0002 3/3 Running 0 4m7s
couchbase-operator-79664dfb6f-9blfw 1/1 Running 0 6m39s
couchbase-operator-admission-74c76787d6-cnr4h 1/1 Running 0 46m
pillowfight-c67nr 0/1 Completed 0 108s
sync-gateway-7c7bcd6b6d-k4qhp 1/1 Running 0 86s
sync-gateway-7c7bcd6b6d-t76kd 1/1 Running 0 86s
- Save the Bucket's topology using cao tool.
- Delete the default bucket through the command:
Prateeks-MacBook-Pro:bin prateekkumar$ kubectl delete couchbasebuckets default
couchbasebucket.couchbase.com "default" deleted
- Edit cbc to set buckets.managed to false and increase DataQuota to 512 MB and Index Quota to 512 MB.
- Load travel-sample bucket. Wait for the sample bucket and it's indexes to load completely.
- Restore the saved topology
Data topology solution:
/
├── default (bucket) create
│ └── _default (scope) create
└── travel-sample (bucket) retain
├── _default (scope) retain
├── inventory (scope) retain
│ ├── airline (collection) retain
│ ├── airport (collection) retain
│ ├── hotel (collection) retain
│ ├── landmark (collection) retain
│ └── route (collection) retain
├── tenant_agent_00 (scope) retain
│ ├── bookings (collection) retain
│ └── users (collection) retain
├── tenant_agent_01 (scope) retain
│ ├── bookings (collection) retain
│ └── users (collection) retain
├── tenant_agent_02 (scope) retain
│ ├── bookings (collection) retain
│ └── users (collection) retain
├── tenant_agent_03 (scope) retain
│ ├── bookings (collection) retain
│ └── users (collection) retain
└── tenant_agent_04 (scope) retain
├── bookings (collection) retain
└── users (collection) retain
WARNING! resources marked as delete may result in data loss.
OK to proceed? (y/N) y
couchbasebuckets/bucket-6f9bf5c5-579e-48a8-8b71-049bd6c22fa6 created
couchbasebuckets/bucket-c821b678-cd9c-4799-bbd3-929c41643eed created
couchbasescopes/scope-cf3445b4-69a6-41a2-862c-06324fefaaa0 created
couchbasescopes/scope-0c907be2-9e00-47f6-9ef2-923c58735363 created
couchbasescopes/scope-089785e2-f519-4d88-a00f-2324b10ca808 created
couchbasescopes/scope-f81508f0-f4f9-482c-810e-ba1f044a01e1 created
couchbasescopes/scope-aec12bfa-9509-4700-8e70-7e0fb6725486 created
couchbasescopes/scope-bb3163be-335b-4374-902f-b754f2d46cf0 created
couchbasecollectiongroups/collectiongroup-f26738b4-7290-4db7-9537-fa712fe72c50 created
couchbasecollectiongroups/collectiongroup-8ad02bb0-5c0e-4c38-b75b-fb6cd3dc4f2b created
couchbasecollectiongroups/collectiongroup-611051bd-88f4-46fa-958c-1921021f6b6c created
couchbasecollectiongroups/collectiongroup-a3fa836b-e9f2-4a05-89e2-a4f5e92ad2a7 created
couchbasecollectiongroups/collectiongroup-67470ec1-b90e-4f9b-bfb7-9644fffdc8b9 created
couchbasecollectiongroups/collectiongroup-a01275fa-2915-4e11-9e81-2ad846565c24 created
couchbasecluster/cb-example updated
Head over to CB UI. We can see ~16-17 items in the default bucket which was deleted.
We can also see ~15 indexes in {}default.{}default.
There was sufficient time gap b/w loading sample-bucket and restoring topology, so I don't know if slow cache clearing is the issue or deletion of bucket not getting properly registered by SGW deployments is.
[Although, I have marked the fix version as 2.3.0, I have strong suspicions on SGW and in which case I'll remove the fix version and close out the ticket, so treat it as such]
Operator logs, SGW logs(both pods) and Cluster logs, Screenshots attached.
https://cb-engineering.s3.amazonaws.com/sgwfinal/collectinfo-2022-04-01T165008-ns_1%40cb-example-0000.cb-example.default.svc.zip
https://cb-engineering.s3.amazonaws.com/sgwfinal/collectinfo-2022-04-01T165008-ns_1%40cb-example-0001.cb-example.default.svc.zip
https://cb-engineering.s3.amazonaws.com/sgwfinal/collectinfo-2022-04-01T165008-ns_1%40cb-example-0002.cb-example.default.svc.zip