Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0-beta-2
    • Fix Version/s: 2.1.0
    • Component/s: tools
    • Security Level: Public
    • Labels:
      None
    • Sprint:
      PCI Team - Sprint 5, PCI Team - Sprint 6, PCI Team - Sprint 7

      Description

      Export data as CSV...

      cbtransfer http://HOST:8091 csv: > /tmp/data.csv

      When you try to import it, the CSV importer changes the document

      cbtransfer /tmp/data.csv http://HOST:8091

      So you end up with completely changed data. That is, the CSV file format that the importer expects (CSV fields are turned into JSON field) is different than the CSV file format that the exporter emits (id,flags,expiration,cas,value)

      1. data.csv
        1.07 MB
        Shashank Gupta
      2. new_data.csv
        5.34 MB
        Shashank Gupta
      1. data_csv.png
        218 kB
      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        kzeller kzeller added a comment -

        Hi Bin,

        Please provide the following information so I can finish documenting this feature:

        Size limits for import/export
        Performance consequences, performance compared to other methods
        Errors that can occur
        How to monitor
        When to use
        Expected data formats
        Sample of command running and output
        Where should command be run from?

        Regards,

        Karen

        Show
        kzeller kzeller added a comment - Hi Bin, Please provide the following information so I can finish documenting this feature: Size limits for import/export Performance consequences, performance compared to other methods Errors that can occur How to monitor When to use Expected data formats Sample of command running and output Where should command be run from? Regards, Karen
        Hide
        bcui Bin Cui added a comment -

        Cbtransfer & CSVs, http://www.couchbase.com/issues/browse/MB-7102
        1. Size limits for import/export
        As far as I know, no limit.
        2. Performance consequences, performance compared to other methods
        Since we reuse same transfer engine and code base, essentially no difference compared to other methods.
        3. Errors that can occur
        1. When csv file is not well formatted.
        4. How to monitor
        1. For csv output, progressing messages on command window
        2. For csv import, you can monitor from admin console as other restore tools.
        5. When to use
        1. Need data transferring between couchbase and other database systems. CSV is a well known data format.
        6. Expected data formats
        1. Standard csv data format
        7. Sample of command running and output
        Export to csv file:
        c:\t1\bin>cbtransfer http://localhost:8091 csv:c:\temp\data.csv -b default -u Ad
        ministrator -p 123456
        ................................................................................
        ................................................................................
        ....................................................................
        ################### 100.0% (1000/1000 msgs)
        bucket: default, msgs transferred...
        : total | last | per sec
        batch : 1143 | 1143 | 280.8
        byte : 10000 | 10000 | 2456.4
        msg : 1000 | 1000 | 245.6
        2013-06-19 14:34:16,209: mt warning: cannot save bucket design on a CSV destination
        done

        Import from csv file:
        c:\t1\bin>cbtransfer c:\temp\data.csv http://localhost:8091 -B default -u Administrator -p 123456
        .
        bucket: data.csv, msgs transferred...
        : total | last | per sec
        batch : 1 | 1 | 0.6
        byte : 5000 | 5000 | 3060.0
        msg : 1000 | 1000 | 612.0
        done

        8. Where should command be run from?
        cbtransfer sits in the same bin directory as cbbackup, cbrestore.

        Show
        bcui Bin Cui added a comment - Cbtransfer & CSVs, http://www.couchbase.com/issues/browse/MB-7102 1. Size limits for import/export As far as I know, no limit. 2. Performance consequences, performance compared to other methods Since we reuse same transfer engine and code base, essentially no difference compared to other methods. 3. Errors that can occur 1. When csv file is not well formatted. 4. How to monitor 1. For csv output, progressing messages on command window 2. For csv import, you can monitor from admin console as other restore tools. 5. When to use 1. Need data transferring between couchbase and other database systems. CSV is a well known data format. 6. Expected data formats 1. Standard csv data format 7. Sample of command running and output Export to csv file: c:\t1\bin>cbtransfer http://localhost:8091 csv:c:\temp\data.csv -b default -u Ad ministrator -p 123456 ................................................................................ ................................................................................ .................................................................... ################### 100.0% (1000/1000 msgs) bucket: default, msgs transferred... : total | last | per sec batch : 1143 | 1143 | 280.8 byte : 10000 | 10000 | 2456.4 msg : 1000 | 1000 | 245.6 2013-06-19 14:34:16,209: mt warning: cannot save bucket design on a CSV destination done Import from csv file: c:\t1\bin>cbtransfer c:\temp\data.csv http://localhost:8091 -B default -u Administrator -p 123456 . bucket: data.csv, msgs transferred... : total | last | per sec batch : 1 | 1 | 0.6 byte : 5000 | 5000 | 3060.0 msg : 1000 | 1000 | 612.0 done 8. Where should command be run from? cbtransfer sits in the same bin directory as cbbackup, cbrestore.
        Hide
        kzeller kzeller added a comment -

        Bin,

        Based on what I have seen in review comments, we need more detail in your answers. See below:

        Cbtransfer & CSVs, http://www.couchbase.com/issues/browse/MB-7102
        Size limits for import/export
        As far as I know, no limit.

        > What are the limitations of using this given limited RAM, CPU and disk space on node/cluster/machine? What can break if you upload something 2x RAM on node, for instance?

        >How should someone prepare their cluster and machines export/import in terms of RAM, CPU, Disk space, RAM quota, etc?

        Performance consequences, performance compared to other methods
        Since we reuse same transfer engine and code base, essentially no difference compared to other methods.
        Errors that can occur
        1. When csv file is not well formatted.

        >What is the exact error message that you will see?

        How to monitor
        1. For csv output, progressing messages on command window
        2. For csv import, you can monitor from admin console as other restore tools.

        > For import, where exactly in admin console do you need to look? What UI element do you look at? What should you see?

        >For import and monitoring with 'other restore tools' which tools are these? What should you see?

        When to use
        1. Need data transferring between couchbase and other database systems. CSV is a well known data format.

        Expected data formats
        1. Standard csv data format

        Sample of command running and output
        In MB-7102, sample command running and output is documented already.
        Where should command be run from?
        cbtransfer sits in the same bin directory as cbbackup, cbrestore.

        >People want to know what machine you run it on (node in cluster, machine outside the cluster, etc.)

        We will need this by COB Thursday, June 20 in order to add it to the documentation by release.

        Thanks,

        Karen

        Show
        kzeller kzeller added a comment - Bin, Based on what I have seen in review comments, we need more detail in your answers. See below: Cbtransfer & CSVs, http://www.couchbase.com/issues/browse/MB-7102 Size limits for import/export As far as I know, no limit. > What are the limitations of using this given limited RAM, CPU and disk space on node/cluster/machine? What can break if you upload something 2x RAM on node, for instance? >How should someone prepare their cluster and machines export/import in terms of RAM, CPU, Disk space, RAM quota, etc? Performance consequences, performance compared to other methods Since we reuse same transfer engine and code base, essentially no difference compared to other methods. Errors that can occur 1. When csv file is not well formatted. >What is the exact error message that you will see? How to monitor 1. For csv output, progressing messages on command window 2. For csv import, you can monitor from admin console as other restore tools. > For import, where exactly in admin console do you need to look? What UI element do you look at? What should you see? >For import and monitoring with 'other restore tools' which tools are these? What should you see? When to use 1. Need data transferring between couchbase and other database systems. CSV is a well known data format. Expected data formats 1. Standard csv data format Sample of command running and output In MB-7102 , sample command running and output is documented already. Where should command be run from? cbtransfer sits in the same bin directory as cbbackup, cbrestore. >People want to know what machine you run it on (node in cluster, machine outside the cluster, etc.) We will need this by COB Thursday, June 20 in order to add it to the documentation by release. Thanks, Karen
        Hide
        kzeller kzeller added a comment - - edited

        from Anil => should run this cbtransfer and tools based on it from outside of the cluster as it will consume resources......??

        Bin => this information should not be specific to the csv scenario. Should be a generic section about CPU consumption, disk size, recommended configuration for cbtransfer.

        Show
        kzeller kzeller added a comment - - edited from Anil => should run this cbtransfer and tools based on it from outside of the cluster as it will consume resources......?? Bin => this information should not be specific to the csv scenario. Should be a generic section about CPU consumption, disk size, recommended configuration for cbtransfer.
        Hide
        kzeller kzeller added a comment -

        Closing per Dipti and Anil.

        Ticket for general cbtransfer guidelines MB-8515

        Show
        kzeller kzeller added a comment - Closing per Dipti and Anil. Ticket for general cbtransfer guidelines MB-8515

          People

          • Assignee:
            bcui Bin Cui
            Reporter:
            steve Steve Yen
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Agile

                Gerrit Reviews

                There are no open Gerrit changes