Uploaded image for project: 'Couchbase C client library libcouchbase'
  1. Couchbase C client library libcouchbase
  2. CCBC-1188

pillowfight ops succeeding against non existent collection

    XMLWordPrintable

Details

    Description

      I pulled down latest libcouchbase repo, built pillowfight according to repo docs. Then I created cluster with 1 bucket and 1 collection. I executed pillowfight against incorrect collection name but ops still made it to KV.

      admins-mbp-98:bin korrigan.clark$ ./cbc version
      cbc:
        Runtime: Version=3.0.0_18_gc3179958d4, Changeset=c3179958d4db71ee70243a1f8470edbf9e405f08
        Headers: Version=3.0.0_18_gc3179958d4, Changeset=c3179958d4db71ee70243a1f8470edbf9e405f08
        Build Timestamp: 2020-03-05 19:57:48
        IO: Default=libevent, Current=libevent, Accessible=libevent,libuv,libev,select
        SSL Runtime: OpenSSL 1.1.1d  10 Sep 2019
        SSL Headers: OpenSSL 1.1.1d  10 Sep 2019
        Snappy: 1.1.1
        Tracing: SUPPORTED
        System: Darwin-19.2.0; x86_64
        CC: Clang 11.0.0.11000033;  -fno-strict-aliasing -ggdb3 -pthread
        CXX: Clang 11.0.0.11000033;  -fno-strict-aliasing -ggdb3 -pthread

      You can see that the only non-default collection is named "collection1"

      admins-mbp-98:bin korrigan.clark$ curl -u Administrator:password -X GET http://172.23.133.13:8091/pools/default/buckets/bucket-1/collections
      {"uid":"2","scopes":[{"name":"scope1","uid":"8","collections":[{"name":"collection1","uid":"8"}]},{"name":"_default","uid":"0","collections":[{"name":"_default","uid":"0"}]}]}

      Pillowfight command, you can see I passed in collection named "csdcadc" which doesnt exist

      ./cbc-pillowfight --password password --batch-size 1000 --num-items 20000000 --num-threads 50 --min-size 512 --max-size 512 --persist-to 0 --replicate-to 0 --collection csdcadc --json --spec "couchbase://172.23.133.13/bucket-1?ipv6=allow&enable_tracing=false" --populate-only Populating using 400 cycles Running. Press Ctrl-C to terminate... ^CS/SEC:      49999
      

      Uploaded screen shot so items in my bucket.

      [root@172-23-133-13 ~]# /opt/couchbase/bin/cbstats -u Administrator -p password -b bucket-1 localhost:11210 collections 
      collection:0x0:items:          139799 
      collection:0x8:items:          0 
      manifest:collection:0x0:name:  _default 
      manifest:collection:0x0:scope: 0x0 
      manifest:collection:0x8:name:  collection1 
      manifest:collection:0x8:scope: 0x8 
      manifest:collections:          2 
      manifest:default_exists:       true 
      manifest:uid:                  2
      

      Attachments

        1. pf.log
          57 kB
        2. pfnot-there.log
          31.20 MB
        3. Screen Shot 2020-03-06 at 10.51.06 AM.png
          Screen Shot 2020-03-06 at 10.51.06 AM.png
          162 kB
        4. Screen Shot 2020-03-25 at 10.16.50 AM.png
          Screen Shot 2020-03-25 at 10.16.50 AM.png
          181 kB
        For Gerrit Dashboard: CCBC-1188
        # Subject Branch Project Status CR V

        Activity

          Further testing shows that pillowfight does not actually write anything to collections. All items go to default collection even if other collection is specified

          korrigan.clark Korrigan Clark added a comment - Further testing shows that pillowfight does not actually write anything to collections. All items go to default collection even if other collection is specified

          The problem in cbc-pillowfight itself, that it does not support collections. It going to be fixed in next release

          avsej Sergey Avseyev added a comment - The problem in cbc-pillowfight itself, that it does not support collections. It going to be fixed in next release

          Sergey Avseyev

          Can you give us a rough ETA on when this will be fixed and release so I can plan my work accordingly. I am working on first round of collections tests currently but have some other test I can implement in the mean time

          korrigan.clark Korrigan Clark added a comment - Sergey Avseyev Can you give us a rough ETA on when this will be fixed and release so I can plan my work accordingly. I am working on first round of collections tests currently but have some other test I can implement in the mean time
          avsej Sergey Avseyev added a comment - - edited

          http://review.couchbase.org/123976

          with this patch cbc-pillowfight will cycle through the collections specified with --collection switch. Collections must be in full form (--collection scope_name.collection_name)

          avsej Sergey Avseyev added a comment - - edited http://review.couchbase.org/123976 with this patch cbc-pillowfight will cycle through the collections specified with --collection switch. Collections must be in full form ( --collection scope_name.collection_name )

          I tried the lastest code and there are two issues. First issue is that ops against a non-existant collection still succeed. Second issue is that pillowifhgt does not fully respect the num-items parameter, it loads more items than specified.

          You can see here that there is default collection and 3 user created collections:

          /opt/couchbase/bin/cbstats -u Administrator -p password -b bucket-1 localhost:11210 collections
           _default:_default:id:    0x0
           collection:0x0:items:    0
           collection:0x8:items:    0
           collection:0x9:items:    0
           collection:0xa:items:    0
           collections:             4
           default_exists:          true
           scope-1:collection-1:id: 0x8
           scope-1:collection-2:id: 0x9
           scope-1:collection-3:id: 0xa
           uid:                     4
          

          I then ran this command, which should load 12,000 items across 3 different collections:

          ./cbc-pillowfight --password password --batch-size 1000 --num-items 12000 --num-threads 5 --min-size 512 --max-size 512 --persist-to 0 --replicate-to 0 --json --spec "couchbase://172.23.96.142/bucket-1?ipv6=allow&enable_tracing=false" --set-pct 0 --num-cycles 1 --collection scope-1.collection-1 --collection scope-1.collection-2 --collection scope-1.collection-3 2>pflog.t.log 
          

          This command does not load exactly 12,000 items, but it does load them into the specified collections evenly as you can see here:

          [root@localhost ~]# /opt/couchbase/bin/cbstats -u Administrator -p password -b bucket-1 localhost:11210 collections
           _default:_default:id:    0x0
           collection:0x0:items:    0
           collection:0x8:items:    1008
           collection:0x9:items:    1000
           collection:0xa:items:    992
           collections:             4
           default_exists:          true
           scope-1:collection-1:id: 0x8
           scope-1:collection-2:id: 0x9
           scope-1:collection-3:id: 0xa
           uid:                     4
          

          Cluster is a 4 node cluster, so we are looking at 1/4th of the vbuckets here. Will upload UI pic as well.

          After flushing the bucket I ran pillowifhgt against a non-existant collection with the following:

          ./cbc-pillowfight --password password --batch-size 1000 --num-items 12000 --num-threads 2 --min-size 512 --max-size 512 --persist-to 0 --replicate-to 0 --json --spec "couchbase://172.23.96.142/bucket-1?ipv6=allow&enable_tracing=false" --set-pct 0 --collection scope-not-there.collection-not-there 2>pfnot-there.log 

          And you can see that all item are loaded in default collection:

          [root@localhost ~]# /opt/couchbase/bin/cbstats -u Administrator -p password -b bucket-1 localhost:11210 collections
           _default:_default:id:    0x0
           collection:0x0:items:    3000
           collection:0x8:items:    0
           collection:0x9:items:    0
           collection:0xa:items:    0
           collections:             4
           default_exists:          true
           scope-1:collection-1:id: 0x8
           scope-1:collection-2:id: 0x9
           scope-1:collection-3:id: 0xa
           uid:                     4
          

          Sergey Avseyev

          korrigan.clark Korrigan Clark added a comment - I tried the lastest code and there are two issues. First issue is that ops against a non-existant collection still succeed. Second issue is that pillowifhgt does not fully respect the num-items parameter, it loads more items than specified. You can see here that there is default collection and 3 user created collections: /opt/couchbase/bin/cbstats -u Administrator -p password -b bucket- 1 localhost: 11210 collections _default:_default:id:    0x0 collection: 0x0 :items:    0 collection: 0x8 :items:    0 collection: 0x9 :items:    0 collection: 0xa :items:    0 collections:             4 default_exists:          true scope- 1 :collection- 1 :id: 0x8 scope- 1 :collection- 2 :id: 0x9 scope- 1 :collection- 3 :id: 0xa uid:                     4 I then ran this command, which should load 12,000 items across 3 different collections: ./cbc-pillowfight --password password --batch-size 1000 --num-items 12000 --num-threads 5 --min-size 512 --max-size 512 --persist-to 0 --replicate-to 0 --json --spec "couchbase://172.23.96.142/bucket-1?ipv6=allow&enable_tracing=false" --set-pct 0 --num-cycles 1 --collection scope- 1 .collection- 1 --collection scope- 1 .collection- 2 --collection scope- 1 .collection- 3 2 >pflog.t.log This command does not load exactly 12,000 items, but it does load them into the specified collections evenly as you can see here: [root @localhost ~]# /opt/couchbase/bin/cbstats -u Administrator -p password -b bucket- 1 localhost: 11210 collections _default:_default:id:    0x0 collection: 0x0 :items:    0 collection: 0x8 :items:    1008 collection: 0x9 :items:    1000 collection: 0xa :items:    992 collections:             4 default_exists:          true scope- 1 :collection- 1 :id: 0x8 scope- 1 :collection- 2 :id: 0x9 scope- 1 :collection- 3 :id: 0xa uid:                     4 Cluster is a 4 node cluster, so we are looking at 1/4th of the vbuckets here. Will upload UI pic as well. After flushing the bucket I ran pillowifhgt against a non-existant collection with the following: ./cbc-pillowfight --password password --batch-size 1000 --num-items 12000 --num-threads 2 --min-size 512 --max-size 512 --persist-to 0 --replicate-to 0 --json --spec "couchbase://172.23.96.142/bucket-1?ipv6=allow&enable_tracing=false" --set-pct 0 --collection scope-not-there.collection-not-there 2 >pfnot-there.log And you can see that all item are loaded in default collection: [root @localhost ~]# /opt/couchbase/bin/cbstats -u Administrator -p password -b bucket- 1 localhost: 11210 collections _default:_default:id:    0x0 collection: 0x0 :items:    3000 collection: 0x8 :items:    0 collection: 0x9 :items:    0 collection: 0xa :items:    0 collections:             4 default_exists:          true scope- 1 :collection- 1 :id: 0x8 scope- 1 :collection- 2 :id: 0x9 scope- 1 :collection- 3 :id: 0xa uid:                     4 Sergey Avseyev

          Hi Korrigan Clark, thanks for the verification. I've fixed fallback to default collection in CCBC-1206

          Waiting for jenkins to verify commits and then I can proceed with the release

          avsej Sergey Avseyev added a comment - Hi Korrigan Clark , thanks for the verification. I've fixed fallback to default collection in CCBC-1206 Waiting for jenkins to verify commits and then I can proceed with the release

          Sergey Avseyev just verified and it works properly now.

          korrigan.clark Korrigan Clark added a comment - Sergey Avseyev  just verified and it works properly now.

          Build couchbase-server-7.0.0-3115 contains libcouchbase commit 79e5791 with commit message:
          CCBC-1188: cbc-pillowfight: fixed --collection switch

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-3115 contains libcouchbase commit 79e5791 with commit message: CCBC-1188 : cbc-pillowfight: fixed --collection switch

          People

            avsej Sergey Avseyev
            korrigan.clark Korrigan Clark
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty