Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-21433

Import/export list/json/csv very slow - 5-50 MB/sec

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Major
    • 5.0.0
    • 5.0.0
    • tools
    • 4.7.0-1230
    • Untriaged
    • Centos 64-bit
    • Unknown

    Description

      http://perf.jenkins.couchbase.com/job/leto/4111/console

      06:01:03 [20/Oct/2016 06:01:03] INFO - export into: /workspace/backup/lines.json
      06:01:09 [20/Oct/2016 06:01:09] INFO - Creating a new database: netleto_470-1230_d5d1722310030
      06:01:11 [20/Oct/2016 06:01:11] INFO - Creating a new database: ns_serverleto_470-1230_d5dbucket-11722310029
      06:01:12 [20/Oct/2016 06:01:12] INFO - Creating a new database: active_tasksleto_470-1230_d5dbucket-1
      06:01:14 [20/Oct/2016 06:01:14] INFO - Creating a new database: atopleto_470-1230_d5d1722310031
      06:01:15 [20/Oct/2016 06:01:15] INFO - Creating a new database: iostatleto_470-1230_d5d1722310031
      06:01:18 [20/Oct/2016 06:01:18] INFO - Creating a new database: netleto_470-1230_d5d1722310031
      06:01:18 [20/Oct/2016 06:01:18] INFO - Creating a new database: ns_serverleto_470-1230_d5dbucket-11722310030
      06:01:21 [20/Oct/2016 06:01:21] INFO - Creating a new database: atopleto_470-1230_d5d1722310032
      06:01:21 [20/Oct/2016 06:01:21] INFO - Creating a new database: iostatleto_470-1230_d5d1722310032
      06:01:24 [20/Oct/2016 06:01:24] INFO - Creating a new database: netleto_470-1230_d5d1722310032
      06:01:25 [20/Oct/2016 06:01:25] INFO - Creating a new database: ns_serverleto_470-1230_d5dbucket-11722310031
      06:01:27 [20/Oct/2016 06:01:27] INFO - Creating a new database: atopleto_470-1230_d5d1722310029
      06:01:28 [20/Oct/2016 06:01:28] INFO - Creating a new database: iostatleto_470-1230_d5d1722310029
      06:01:31 [20/Oct/2016 06:01:31] INFO - Creating a new database: netleto_470-1230_d5d1722310029
      06:01:31 [20/Oct/2016 06:01:31] INFO - Creating a new database: ns_serverleto_470-1230_d5dbucket-11722310032
      06:01:55 [20/Oct/2016 06:01:55] WARNING - {"error":"Bad Request","reason":"Error reading body: read tcp 172.23.100.213:50854: i/o timeout"}
      06:01:56 [20/Oct/2016 06:01:56] WARNING - [Errno 104] Connection reset by peer
      06:10:12 [20/Oct/2016 06:10:12] INFO - Export completed in 558.0 sec, Export size is 57.0 GB
      06:10:12 [20/Oct/2016 06:10:12] INFO - Terminating stats collectors
      06:10:12 [20/Oct/2016 06:10:12] INFO - Reconstructing measurements
      06:10:12 [20/Oct/2016 06:10:12] INFO - Adding snapshot: leto_470-1230_d5d_export
      06:10:12 [20/Oct/2016 06:10:12] INFO - http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=leto_470-1230_d5d_export
      06:10:25 [20/Oct/2016 06:10:25] INFO - Successfully posted: Export_JSON_Lines_tools_expimp_60M_json_lines_leto, {
      06:10:25     "cluster": "leto", 
      06:10:25     "larger_is_better": "true", 
      06:10:25     "title": "Export JSON Lines (Avg. MB/sec), 1 bucket x 60M x 1KB, Idle"
      06:10:25 }
      06:10:25 [20/Oct/2016 06:10:25] INFO - Successfully posted: leto, {
      06:10:25     "CPU": "E5-2630 (24 vCPU)", 
      06:10:25     "Disk": "SSD", 
      06:10:25     "Memory": "64 GB", 
      06:10:25     "OS": "CentOS 7", 
      06:10:25     "Platform": "HW"
      06:10:25 }
      06:10:25 [20/Oct/2016 06:10:25] INFO - Successfully posted: {
      06:10:25     "build": "4.7.0-1230", 
      06:10:25     "build_url": "http://perf.jenkins.couchbase.com/job/leto/4111/", 
      06:10:25     "metric": "Export_JSON_Lines_tools_expimp_60M_json_lines_leto", 
      06:10:25     "snapshots": [
      06:10:25         "leto_470-1230_d5d_export"
      06:10:25     ], 
      06:10:25     "value": 105.0
      06:10:25 }
      06:10:25 [20/Oct/2016 06:10:25] INFO - Flushing bucket: bucket-1
      06:10:34 [20/Oct/2016 06:10:34] INFO - Starting stats collectors
      06:10:34 [20/Oct/2016 06:10:34] INFO - import from: /workspace/backup/lines.json
      08:05:16 [20/Oct/2016 08:05:16] INFO - Import completed in 6882.2 sec, Import size is 57.0 GB
      08:05:16 [20/Oct/2016 08:05:16] INFO - Terminating stats collectors
      08:05:16 [20/Oct/2016 08:05:16] INFO - Reconstructing measurements
      08:05:16 [20/Oct/2016 08:05:16] INFO - Adding snapshot: leto_470-1230_d5d_import_data
      08:05:16 [20/Oct/2016 08:05:16] INFO - http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=leto_470-1230_d5d_import_data
      08:06:37 [20/Oct/2016 08:06:37] INFO - Successfully posted: Import_JSON_Lines_tools_expimp_60M_json_lines_leto, {
      08:06:37     "cluster": "leto", 
      08:06:37     "larger_is_better": "true", 
      08:06:37     "title": "Import JSON Lines (Avg. MB/sec), 1 bucket x 60M x 1KB, Idle"
      08:06:37 }
      08:06:37 [20/Oct/2016 08:06:37] INFO - Successfully posted: leto, {
      08:06:37     "CPU": "E5-2630 (24 vCPU)", 
      08:06:37     "Disk": "SSD", 
      08:06:37     "Memory": "64 GB", 
      08:06:37     "OS": "CentOS 7", 
      08:06:37     "Platform": "HW"
      08:06:37 }
      08:06:38 [20/Oct/2016 08:06:38] INFO - Successfully posted: {
      08:06:38     "build": "4.7.0-1230", 
      08:06:38     "build_url": "http://perf.jenkins.couchbase.com/job/leto/4111/", 
      08:06:38     "metric": "Import_JSON_Lines_tools_expimp_60M_json_lines_leto", 
      08:06:38     "snapshots": [
      08:06:38         "leto_470-1230_d5d_export", 
      08:06:38         "leto_470-1230_d5d_import_data"
      08:06:38     ], 
      08:06:38     "value": 9.0
      08:06:38 }
      

      Attachments

        Issue Links

          For Gerrit Dashboard: MB-21433
          # Subject Branch Project Status CR V

          Activity

            People

              mikew Mike Wiederhold [X] (Inactive)
              andreibaranouski Andrei Baranouski
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty