Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-45300

Slow data load on newly created bucket if cbimport is run immediately

    XMLWordPrintable

Details

    • Bug
    • Resolution: Unresolved
    • Minor
    • Morpheus
    • Cheshire-Cat
    • tools
    • None
    • Untriaged
    • 1
    • Unknown

    Description

      slow data load on newly created bucket if cbimport is run immediately after the bucket is created.   

      This is a bad developer experience, one would assume providing the --wait flag that the bucket would be ready.

      The below cbimport takes 53.3 sec.

       

      couchbase-cli bucket-create -c localhost:8091 \
        -u $CB_USERNAME -p $CB_PASSWORD  \
        --bucket=vb${inst}  \
        --bucket-type=couchbase  \
        --bucket-ramsize=100  \
        --bucket-replica=1  \
        --enable-flush=1  \
        --wait
       
      # no sleep run cbimport as soon as possible 
       
      cbimport json -c couchbase://127.0.0.1 -u $CB_USERNAME -p $CB_PASSWORD -b vb01 -d file:///tmp/data.json -f lines -t 2 -g evtest0::%id%
      JSON `file:///tmp/data.json` imported to `couchbase://127.0.0.1` successfully
      Documents imported: 3000 Documents failed: 0
       
      real	0m53.357s
      user	0m0.306s
      sys	0m0.180s

      adding just a 10 second sleep before the cbimport it runs 10X faster taking only 5.1 sec.

       

      couchbase-cli bucket-create -c localhost:8091 \
        -u $CB_USERNAME -p $CB_PASSWORD  \
        --bucket=vb01  \
        --bucket-type=couchbase  \
        --bucket-ramsize=100  \
        --bucket-replica=1  \
        --enable-flush=1  \
        --wait
       
      sleep 10
       
      time cbimport json -c couchbase://127.0.0.1 \
        -u $CB_USERNAME -p $CB_PASSWORD -b vb01 \
        -d file:///tmp/data.json -f lines -t 2 -g evtest0::%id%
      JSON `file:///tmp/data.json` imported to `couchbase://127.0.0.1` successfully
      Documents imported: 3000 Documents failed: 0
       
      real	0m5.152s
      user	0m0.146s
      sys	0m0.052s

       

       

       

       

       

       

      SUCCESS: Bucket created

      add 3000 docs to vb01

      gen/cbimport block: 1 of size: 3000 start at Sat Mar 27 21:57:30 UTC 2021

      3000 /tmp/data.json

      JSON `file:///tmp/data.json` imported to `couchbase://127.0.0.1` successfully

      Documents imported: 3000 Documents failed: 0

       

      real 0m53.500s

      user 0m0.345s

      sys 0m0.219s

      [ec2-user@ip-10-0-0-187 scripts]$ ./test_del_bucket_gen.sh admin jtester 1

      SUCCESS: Bucket deleted

      [ec2-user@ip-10-0-0-187 scripts]$ !vi

      vi test_add_bucket_gen.sh

      [ec2-user@ip-10-0-0-187 scripts]$

      [ec2-user@ip-10-0-0-187 scripts]$

      [ec2-user@ip-10-0-0-187 scripts]$

      [ec2-user@ip-10-0-0-187 scripts]$ ./test_add_bucket_gen.sh admin jtester 1 3000

      SUCCESS: Bucket created

      add 3000 docs to vb01

      gen/cbimport block: 1 of size: 3000 start at Sat Mar 27 22:00:00 UTC 2021

      3000 /tmp/data.json

      JSON `file:///tmp/data.json` imported to `couchbase://127.0.0.1` successfully

      Documents imported: 3000 Documents failed: 0

       

      real 0m5.119s

      user 0m0.143s

      sys 0m0.048s

      [ec2-user@ip-10-0-0-187 scripts]$ !vi

      Attachments

        1. collectinfo-2021-03-30T091352-n_0@cb.local.zip
          5.97 MB
        2. collectinfo-2021-03-30T092656-n_0@cb.local.zip
          6.03 MB
        3. test.json
          10.34 MB
        4. test.sh
          0.7 kB
        5. WORK.zip
          1 kB

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              owend Daniel Owen
              jon.strabala Jon Strabala
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty