Uploaded image for project: 'Couchbase C client library libcouchbase'
  1. Couchbase C client library libcouchbase
  2. CCBC-1188

pillowfight ops succeeding against non existent collection

    XMLWordPrintable

Details

    Description

      I pulled down latest libcouchbase repo, built pillowfight according to repo docs. Then I created cluster with 1 bucket and 1 collection. I executed pillowfight against incorrect collection name but ops still made it to KV.

      admins-mbp-98:bin korrigan.clark$ ./cbc version
      cbc:
        Runtime: Version=3.0.0_18_gc3179958d4, Changeset=c3179958d4db71ee70243a1f8470edbf9e405f08
        Headers: Version=3.0.0_18_gc3179958d4, Changeset=c3179958d4db71ee70243a1f8470edbf9e405f08
        Build Timestamp: 2020-03-05 19:57:48
        IO: Default=libevent, Current=libevent, Accessible=libevent,libuv,libev,select
        SSL Runtime: OpenSSL 1.1.1d  10 Sep 2019
        SSL Headers: OpenSSL 1.1.1d  10 Sep 2019
        Snappy: 1.1.1
        Tracing: SUPPORTED
        System: Darwin-19.2.0; x86_64
        CC: Clang 11.0.0.11000033;  -fno-strict-aliasing -ggdb3 -pthread
        CXX: Clang 11.0.0.11000033;  -fno-strict-aliasing -ggdb3 -pthread

      You can see that the only non-default collection is named "collection1"

      admins-mbp-98:bin korrigan.clark$ curl -u Administrator:password -X GET http://172.23.133.13:8091/pools/default/buckets/bucket-1/collections
      {"uid":"2","scopes":[{"name":"scope1","uid":"8","collections":[{"name":"collection1","uid":"8"}]},{"name":"_default","uid":"0","collections":[{"name":"_default","uid":"0"}]}]}

      Pillowfight command, you can see I passed in collection named "csdcadc" which doesnt exist

      ./cbc-pillowfight --password password --batch-size 1000 --num-items 20000000 --num-threads 50 --min-size 512 --max-size 512 --persist-to 0 --replicate-to 0 --collection csdcadc --json --spec "couchbase://172.23.133.13/bucket-1?ipv6=allow&enable_tracing=false" --populate-only Populating using 400 cycles Running. Press Ctrl-C to terminate... ^CS/SEC:      49999
      

      Uploaded screen shot so items in my bucket.

      [root@172-23-133-13 ~]# /opt/couchbase/bin/cbstats -u Administrator -p password -b bucket-1 localhost:11210 collections 
      collection:0x0:items:          139799 
      collection:0x8:items:          0 
      manifest:collection:0x0:name:  _default 
      manifest:collection:0x0:scope: 0x0 
      manifest:collection:0x8:name:  collection1 
      manifest:collection:0x8:scope: 0x8 
      manifest:collections:          2 
      manifest:default_exists:       true 
      manifest:uid:                  2
      

      Attachments

        1. pf.log
          57 kB
        2. pfnot-there.log
          31.20 MB
        3. Screen Shot 2020-03-06 at 10.51.06 AM.png
          Screen Shot 2020-03-06 at 10.51.06 AM.png
          162 kB
        4. Screen Shot 2020-03-25 at 10.16.50 AM.png
          Screen Shot 2020-03-25 at 10.16.50 AM.png
          181 kB
        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          Further testing shows that pillowfight does not actually write anything to collections. All items go to default collection even if other collection is specified

          korrigan.clark Korrigan Clark added a comment - Further testing shows that pillowfight does not actually write anything to collections. All items go to default collection even if other collection is specified

          The problem in cbc-pillowfight itself, that it does not support collections. It going to be fixed in next release

          avsej Sergey Avseyev added a comment - The problem in cbc-pillowfight itself, that it does not support collections. It going to be fixed in next release

          Sergey Avseyev

          Can you give us a rough ETA on when this will be fixed and release so I can plan my work accordingly. I am working on first round of collections tests currently but have some other test I can implement in the mean time

          korrigan.clark Korrigan Clark added a comment - Sergey Avseyev Can you give us a rough ETA on when this will be fixed and release so I can plan my work accordingly. I am working on first round of collections tests currently but have some other test I can implement in the mean time
          avsej Sergey Avseyev added a comment - - edited

          http://review.couchbase.org/123976

          with this patch cbc-pillowfight will cycle through the collections specified with --collection switch. Collections must be in full form (--collection scope_name.collection_name)

          avsej Sergey Avseyev added a comment - - edited http://review.couchbase.org/123976 with this patch cbc-pillowfight will cycle through the collections specified with --collection switch. Collections must be in full form ( --collection scope_name.collection_name )

          I tried the lastest code and there are two issues. First issue is that ops against a non-existant collection still succeed. Second issue is that pillowifhgt does not fully respect the num-items parameter, it loads more items than specified.

          You can see here that there is default collection and 3 user created collections:

          /opt/couchbase/bin/cbstats -u Administrator -p password -b bucket-1 localhost:11210 collections
           _default:_default:id:    0x0
           collection:0x0:items:    0
           collection:0x8:items:    0
           collection:0x9:items:    0
           collection:0xa:items:    0
           collections:             4
           default_exists:          true
           scope-1:collection-1:id: 0x8
           scope-1:collection-2:id: 0x9
           scope-1:collection-3:id: 0xa
           uid:                     4
          

          I then ran this command, which should load 12,000 items across 3 different collections:

          ./cbc-pillowfight --password password --batch-size 1000 --num-items 12000 --num-threads 5 --min-size 512 --max-size 512 --persist-to 0 --replicate-to 0 --json --spec "couchbase://172.23.96.142/bucket-1?ipv6=allow&enable_tracing=false" --set-pct 0 --num-cycles 1 --collection scope-1.collection-1 --collection scope-1.collection-2 --collection scope-1.collection-3 2>pflog.t.log 
          

          This command does not load exactly 12,000 items, but it does load them into the specified collections evenly as you can see here:

          [root@localhost ~]# /opt/couchbase/bin/cbstats -u Administrator -p password -b bucket-1 localhost:11210 collections
           _default:_default:id:    0x0
           collection:0x0:items:    0
           collection:0x8:items:    1008
           collection:0x9:items:    1000
           collection:0xa:items:    992
           collections:             4
           default_exists:          true
           scope-1:collection-1:id: 0x8
           scope-1:collection-2:id: 0x9
           scope-1:collection-3:id: 0xa
           uid:                     4
          

          Cluster is a 4 node cluster, so we are looking at 1/4th of the vbuckets here. Will upload UI pic as well.

          After flushing the bucket I ran pillowifhgt against a non-existant collection with the following:

          ./cbc-pillowfight --password password --batch-size 1000 --num-items 12000 --num-threads 2 --min-size 512 --max-size 512 --persist-to 0 --replicate-to 0 --json --spec "couchbase://172.23.96.142/bucket-1?ipv6=allow&enable_tracing=false" --set-pct 0 --collection scope-not-there.collection-not-there 2>pfnot-there.log 

          And you can see that all item are loaded in default collection:

          [root@localhost ~]# /opt/couchbase/bin/cbstats -u Administrator -p password -b bucket-1 localhost:11210 collections
           _default:_default:id:    0x0
           collection:0x0:items:    3000
           collection:0x8:items:    0
           collection:0x9:items:    0
           collection:0xa:items:    0
           collections:             4
           default_exists:          true
           scope-1:collection-1:id: 0x8
           scope-1:collection-2:id: 0x9
           scope-1:collection-3:id: 0xa
           uid:                     4
          

          Sergey Avseyev

          korrigan.clark Korrigan Clark added a comment - I tried the lastest code and there are two issues. First issue is that ops against a non-existant collection still succeed. Second issue is that pillowifhgt does not fully respect the num-items parameter, it loads more items than specified. You can see here that there is default collection and 3 user created collections: /opt/couchbase/bin/cbstats -u Administrator -p password -b bucket- 1 localhost: 11210 collections _default:_default:id:    0x0 collection: 0x0 :items:    0 collection: 0x8 :items:    0 collection: 0x9 :items:    0 collection: 0xa :items:    0 collections:             4 default_exists:          true scope- 1 :collection- 1 :id: 0x8 scope- 1 :collection- 2 :id: 0x9 scope- 1 :collection- 3 :id: 0xa uid:                     4 I then ran this command, which should load 12,000 items across 3 different collections: ./cbc-pillowfight --password password --batch-size 1000 --num-items 12000 --num-threads 5 --min-size 512 --max-size 512 --persist-to 0 --replicate-to 0 --json --spec "couchbase://172.23.96.142/bucket-1?ipv6=allow&enable_tracing=false" --set-pct 0 --num-cycles 1 --collection scope- 1 .collection- 1 --collection scope- 1 .collection- 2 --collection scope- 1 .collection- 3 2 >pflog.t.log This command does not load exactly 12,000 items, but it does load them into the specified collections evenly as you can see here: [root @localhost ~]# /opt/couchbase/bin/cbstats -u Administrator -p password -b bucket- 1 localhost: 11210 collections _default:_default:id:    0x0 collection: 0x0 :items:    0 collection: 0x8 :items:    1008 collection: 0x9 :items:    1000 collection: 0xa :items:    992 collections:             4 default_exists:          true scope- 1 :collection- 1 :id: 0x8 scope- 1 :collection- 2 :id: 0x9 scope- 1 :collection- 3 :id: 0xa uid:                     4 Cluster is a 4 node cluster, so we are looking at 1/4th of the vbuckets here. Will upload UI pic as well. After flushing the bucket I ran pillowifhgt against a non-existant collection with the following: ./cbc-pillowfight --password password --batch-size 1000 --num-items 12000 --num-threads 2 --min-size 512 --max-size 512 --persist-to 0 --replicate-to 0 --json --spec "couchbase://172.23.96.142/bucket-1?ipv6=allow&enable_tracing=false" --set-pct 0 --collection scope-not-there.collection-not-there 2 >pfnot-there.log And you can see that all item are loaded in default collection: [root @localhost ~]# /opt/couchbase/bin/cbstats -u Administrator -p password -b bucket- 1 localhost: 11210 collections _default:_default:id:    0x0 collection: 0x0 :items:    3000 collection: 0x8 :items:    0 collection: 0x9 :items:    0 collection: 0xa :items:    0 collections:             4 default_exists:          true scope- 1 :collection- 1 :id: 0x8 scope- 1 :collection- 2 :id: 0x9 scope- 1 :collection- 3 :id: 0xa uid:                     4 Sergey Avseyev

          Hi Korrigan Clark, thanks for the verification. I've fixed fallback to default collection in CCBC-1206

          Waiting for jenkins to verify commits and then I can proceed with the release

          avsej Sergey Avseyev added a comment - Hi Korrigan Clark , thanks for the verification. I've fixed fallback to default collection in CCBC-1206 Waiting for jenkins to verify commits and then I can proceed with the release

          Sergey Avseyev just verified and it works properly now.

          korrigan.clark Korrigan Clark added a comment - Sergey Avseyev  just verified and it works properly now.

          Build couchbase-server-7.0.0-3115 contains libcouchbase commit 79e5791 with commit message:
          CCBC-1188: cbc-pillowfight: fixed --collection switch

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-3115 contains libcouchbase commit 79e5791 with commit message: CCBC-1188 : cbc-pillowfight: fixed --collection switch

          People

            avsej Sergey Avseyev
            korrigan.clark Korrigan Clark
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty