Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-6170

cbtransfer gen: performance drops after data load phase

    Details

    • Type: Story
    • Status: Closed
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: 2.0, 2.0.1
    • Fix Version/s: 3.0
    • Component/s: tools
    • Security Level: Public
    • Labels:
      None
    • Sprint:
      PCI Team - Sprint 11

      Description

      the "cbtransfer gen: http://HOST:8091" performance (mysteriously) drops after initial data loading phase.

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        bcui Bin Cui (Inactive) added a comment -

        Use python profile, we have the following stats
        python -m cProfile cbworkloadgen

        default: -i = 100000, -t = 1, -s = 10

        23891 function calls (19585 primitive calls) in 102.732 seconds

        Ordered by: standard name

        ncalls tottime percall cumtime percall filename:lineno(function)
        1 0.000 0.000 0.093 0.093 _init_.py:270(loads)
        1 0.000 0.000 102.583 102.583 cbworkloadgen:30(main)
        1 0.000 0.000 0.003 0.003 cbworkloadgen:43(opt_construct)
        1 0.001 0.001 102.732 102.732 cbworkloadgen:6(<module>)
        1 0.000 0.000 0.002 0.002 cbworkloadgen:84(add_parser_options)
        99 0.001 0.000 0.001 0.000 collections.py:277(<genexpr>)
        13 0.000 0.000 0.000 0.000 collections.py:294(<genexpr>)
        1 0.000 0.000 0.000 0.000 collections.py:356(Counter)
        48/1 0.001 0.000 0.002 0.002 copy.py:145(deepcopy)
        38 0.000 0.000 0.000 0.000 copy.py:198(_deepcopy_atomic)
        2/1 0.000 0.000 0.001 0.001 copy.py:253(_deepcopy_dict)
        41 0.000 0.000 0.000 0.000 copy.py:267(_keep_alive)
        1 0.000 0.000 0.002 0.002 copy.py:283(_deepcopy_inst)
        1 0.003 0.003 0.007 0.007 couchstore.py:4(<module>)
        1 0.004 0.004 0.004 0.004 dbapi2.py:24(<module>)
        1 0.000 0.000 0.000 0.000 dbapi2.py:55(register_adapters_and_converters)
        11/1 0.001 0.000 0.093 0.093 decoder.py:148(JSONObject)
        1 0.000 0.000 0.001 0.001 decoder.py:17(_floatconstants)
        1 0.001 0.001 0.010 0.010 decoder.py:2(<module>)
        1029/1 0.025 0.000 0.093 0.093 decoder.py:223(JSONArray)
        1 0.000 0.000 0.093 0.093 decoder.py:330(decode)
        1 0.000 0.000 0.093 0.093 decoder.py:341(raw_decode)
        100 0.002 0.000 0.004 0.000 decoder.py:65(py_scanstring)
        1 0.001 0.001 0.004 0.004 encoder.py:2(<module>)
        1 0.005 0.005 0.062 0.062 httplib.py:67(<module>)
        1 0.000 0.000 0.002 0.002 httplib.py:754(connect)
        1 0.000 0.000 0.002 0.002 httplib.py:772(send)
        1 0.000 0.000 0.002 0.002 httplib.py:799(_send_output)
        1 0.000 0.000 0.001 0.001 httplib.py:820(putrequest)
        3 0.000 0.000 0.000 0.000 httplib.py:930(putheader)
        1 0.000 0.000 0.002 0.002 httplib.py:941(endheaders)
        1 0.000 0.000 0.003 0.003 httplib.py:956(request)
        1 0.000 0.000 0.003 0.003 httplib.py:977(_send_request)
        1 0.000 0.000 0.016 0.016 httplib.py:994(getresponse)
        1 0.001 0.001 0.006 0.006 mc_bin_client.py:6(<module>)
        66 0.001 0.000 0.002 0.000 memcacheConstants.py:113(<genexpr>)
        1 0.001 0.001 0.002 0.002 memcacheConstants.py:5(<module>)
        1 0.003 0.003 0.039 0.039 mimetools.py:1(<module>)
        1 0.000 0.000 0.113 0.113 pump.py:136(check_endpoints)
        1 0.003 0.003 102.459 102.459 pump.py:186(transfer_bucket_msgs)
        1 0.007 0.007 0.096 0.096 pump.py:3(<module>)
        1 0.000 0.000 0.003 0.003 pump.py:35(report)
        1 0.000 0.000 0.019 0.019 pump.py:811(rest_request)
        1 0.000 0.000 0.113 0.113 pump.py:851(rest_request_json)
        1 0.000 0.000 0.113 0.113 pump.py:866(rest_couchbase)
        1 0.000 0.000 102.573 102.573 pump.py:94(run)
        1 0.003 0.003 0.012 0.012 pump_bfd.py:3(<module>)
        1 0.000 0.000 0.000 0.000 pump_bfd.py:50(BFDSource)
        1 0.002 0.002 0.008 0.008 pump_cb.py:3(<module>)
        1 0.000 0.000 0.113 0.113 pump_cb.py:88(check)
        1 0.001 0.001 0.008 0.008 pump_sfd.py:3(<module>)
        1 0.008 0.008 0.148 0.148 pump_transfer.py:3(<module>)
        1 0.000 0.000 102.580 102.580 pump_transfer.py:46(main)
        1 0.000 0.000 0.006 0.006 pump_transfer.py:79(opt_parse)
        1 0.000 0.000 0.003 0.003 pump_transfer.py:91(opt_parser)
        1 0.000 0.000 0.003 0.003 pump_transfer.py:97(opt_parser_options)
        1 0.000 0.000 0.029 0.029 random.py:100(seed)
        1 0.002 0.002 0.032 0.032 random.py:40(<module>)
        1 0.000 0.000 0.029 0.029 random.py:91(_init_)
        9 0.000 0.000 0.014 0.002 re.py:188(compile)
        9 0.000 0.000 0.014 0.002 re.py:226(_compile)
        3150/1 0.035 0.000 0.093 0.093 scanner.py:27(_scan_once)
        1 0.008 0.008 0.008 0.008 socket.py:45(<module>)
        1 0.000 0.000 0.002 0.002 socket.py:537(create_connection)
        1 0.004 0.004 0.036 0.036 tempfile.py:18(<module>)
        485 0.002 0.000 0.003 0.000 urllib.py:1199(<genexpr>)
        485 0.003 0.000 0.004 0.000 urlparse.py:315(<genexpr>)
        38 0.000 0.000 0.000 0.000 {_ctypes.sizeof}
        1768 0.004 0.000 0.004 0.000

        {chr}

        27 0.000 0.000 0.000 0.000

        {filter}

        53 0.000 0.000 0.000 0.000

        {getattr}

        72 0.000 0.000 0.000 0.000

        {globals}

        37 0.000 0.000 0.000 0.000

        {hasattr}

        93 0.000 0.000 0.000 0.000

        {id}

        262 0.001 0.000 0.001 0.000

        {isinstance}

        883/868 0.002 0.000 0.002 0.000

        {len}

        10 0.001 0.000 0.001 0.000

        {method 'acquire' of 'thread.lock' objects}

        11 0.000 0.000 0.000 0.000

        {method 'add' of 'set' objects}

        3866 0.007 0.000 0.007 0.000

        {method 'append' of 'list' objects}

        2180 0.005 0.000 0.005 0.000

        {method 'end' of '_sre.SRE_Match' objects}

        11 0.000 0.000 0.000 0.000

        {method 'extend' of 'list' objects}

        205 0.000 0.000 0.000 0.000

        {method 'find' of 'str' objects}

        191 0.000 0.000 0.000 0.000

        {method 'format' of 'str' objects}

        224 0.000 0.000 0.000 0.000

        {method 'get' of 'dict' objects}

        7 0.000 0.000 0.000 0.000

        {method 'getvalue' of 'cStringIO.StringO' objects}

        2178 0.005 0.000 0.005 0.000

        {method 'groups' of '_sre.SRE_Match' objects}

        86 0.000 0.000 0.000 0.000

        {method 'isalnum' of 'str' objects}

        13 0.000 0.000 0.000 0.000

        {method 'isdigit' of 'str' objects}

        14 0.000 0.000 0.000 0.000

        {method 'items' of 'dict' objects}

        5 0.000 0.000 0.000 0.000

        {method 'iteritems' of 'dict' objects}

        6 0.000 0.000 0.000 0.000

        {method 'iterkeys' of 'dict' objects}

        18 0.000 0.000 0.000 0.000

        {method 'join' of 'str' objects}

        100 0.000 0.000 0.000 0.000

        {method 'join' of 'unicode' objects}

        4 0.000 0.000 0.000 0.000

        {method 'ljust' of 'str' objects}

        24 0.000 0.000 0.000 0.000

        {method 'lower' of 'str' objects}

        2180 0.014 0.000 0.014 0.000

        {method 'match' of '_sre.SRE_Pattern' objects}

        8 0.000 0.000 0.000 0.000

        {method 'pop' of 'dict' objects}

        8 0.000 0.000 0.000 0.000

        {method 'pop' of 'list' objects}

        206 0.012 0.000 0.012 0.000

        {method 'recv' of '_socket.socket' objects}

        6 0.000 0.000 0.000 0.000

        {method 'release' of 'thread.lock' objects}

        7 0.000 0.000 0.000 0.000

        {method 'remove' of 'list' objects}

        43 0.000 0.000 0.000 0.000

        {method 'replace' of 'str' objects}

        12 0.000 0.000 0.000 0.000

        {method 'rjust' of 'str' objects}

        17 0.000 0.000 0.000 0.000

        {method 'seek' of 'cStringIO.StringO' objects}

        32 0.000 0.000 0.000 0.000

        {method 'setdefault' of 'dict' objects}

        23 0.000 0.000 0.000 0.000

        {method 'split' of 'str' objects}

        106 0.000 0.000 0.000 0.000

        {method 'startswith' of 'str' objects}

        17 0.000 0.000 0.000 0.000

        {method 'strip' of 'str' objects}

        16 0.000 0.000 0.000 0.000

        {method 'tell' of 'cStringIO.StringO' objects}

        10 0.000 0.000 0.000 0.000

        {method 'upper' of 'str' objects}

        195 0.000 0.000 0.000 0.000

        {method 'write' of 'cStringIO.StringO' objects}

        7 0.004 0.001 0.004 0.001

        {method 'write' of 'file' objects}

        211 0.001 0.000 0.001 0.000

        {min}

        1 0.029 0.029 0.029 0.029

        {nt.urandom}

        39 0.000 0.000 0.000 0.000

        {ord}

        14 0.000 0.000 0.000 0.000

        {range}

        383 0.001 0.000 0.001 0.000

        {setattr}

        11 0.000 0.000 0.000 0.000

        {thread.allocate_lock}

        11 0.000 0.000 0.000 0.000

        {thread.get_ident}

        149 102.449 0.688 102.449 0.688

        {time.sleep}
        Show
        bcui Bin Cui (Inactive) added a comment - Use python profile, we have the following stats python -m cProfile cbworkloadgen default: -i = 100000, -t = 1, -s = 10 23891 function calls (19585 primitive calls) in 102.732 seconds Ordered by: standard name ncalls tottime percall cumtime percall filename:lineno(function) 1 0.000 0.000 0.093 0.093 _ init _.py:270(loads) 1 0.000 0.000 102.583 102.583 cbworkloadgen:30(main) 1 0.000 0.000 0.003 0.003 cbworkloadgen:43(opt_construct) 1 0.001 0.001 102.732 102.732 cbworkloadgen:6(<module>) 1 0.000 0.000 0.002 0.002 cbworkloadgen:84(add_parser_options) 99 0.001 0.000 0.001 0.000 collections.py:277(<genexpr>) 13 0.000 0.000 0.000 0.000 collections.py:294(<genexpr>) 1 0.000 0.000 0.000 0.000 collections.py:356(Counter) 48/1 0.001 0.000 0.002 0.002 copy.py:145(deepcopy) 38 0.000 0.000 0.000 0.000 copy.py:198(_deepcopy_atomic) 2/1 0.000 0.000 0.001 0.001 copy.py:253(_deepcopy_dict) 41 0.000 0.000 0.000 0.000 copy.py:267(_keep_alive) 1 0.000 0.000 0.002 0.002 copy.py:283(_deepcopy_inst) 1 0.003 0.003 0.007 0.007 couchstore.py:4(<module>) 1 0.004 0.004 0.004 0.004 dbapi2.py:24(<module>) 1 0.000 0.000 0.000 0.000 dbapi2.py:55(register_adapters_and_converters) 11/1 0.001 0.000 0.093 0.093 decoder.py:148(JSONObject) 1 0.000 0.000 0.001 0.001 decoder.py:17(_floatconstants) 1 0.001 0.001 0.010 0.010 decoder.py:2(<module>) 1029/1 0.025 0.000 0.093 0.093 decoder.py:223(JSONArray) 1 0.000 0.000 0.093 0.093 decoder.py:330(decode) 1 0.000 0.000 0.093 0.093 decoder.py:341(raw_decode) 100 0.002 0.000 0.004 0.000 decoder.py:65(py_scanstring) 1 0.001 0.001 0.004 0.004 encoder.py:2(<module>) 1 0.005 0.005 0.062 0.062 httplib.py:67(<module>) 1 0.000 0.000 0.002 0.002 httplib.py:754(connect) 1 0.000 0.000 0.002 0.002 httplib.py:772(send) 1 0.000 0.000 0.002 0.002 httplib.py:799(_send_output) 1 0.000 0.000 0.001 0.001 httplib.py:820(putrequest) 3 0.000 0.000 0.000 0.000 httplib.py:930(putheader) 1 0.000 0.000 0.002 0.002 httplib.py:941(endheaders) 1 0.000 0.000 0.003 0.003 httplib.py:956(request) 1 0.000 0.000 0.003 0.003 httplib.py:977(_send_request) 1 0.000 0.000 0.016 0.016 httplib.py:994(getresponse) 1 0.001 0.001 0.006 0.006 mc_bin_client.py:6(<module>) 66 0.001 0.000 0.002 0.000 memcacheConstants.py:113(<genexpr>) 1 0.001 0.001 0.002 0.002 memcacheConstants.py:5(<module>) 1 0.003 0.003 0.039 0.039 mimetools.py:1(<module>) 1 0.000 0.000 0.113 0.113 pump.py:136(check_endpoints) 1 0.003 0.003 102.459 102.459 pump.py:186(transfer_bucket_msgs) 1 0.007 0.007 0.096 0.096 pump.py:3(<module>) 1 0.000 0.000 0.003 0.003 pump.py:35(report) 1 0.000 0.000 0.019 0.019 pump.py:811(rest_request) 1 0.000 0.000 0.113 0.113 pump.py:851(rest_request_json) 1 0.000 0.000 0.113 0.113 pump.py:866(rest_couchbase) 1 0.000 0.000 102.573 102.573 pump.py:94(run) 1 0.003 0.003 0.012 0.012 pump_bfd.py:3(<module>) 1 0.000 0.000 0.000 0.000 pump_bfd.py:50(BFDSource) 1 0.002 0.002 0.008 0.008 pump_cb.py:3(<module>) 1 0.000 0.000 0.113 0.113 pump_cb.py:88(check) 1 0.001 0.001 0.008 0.008 pump_sfd.py:3(<module>) 1 0.008 0.008 0.148 0.148 pump_transfer.py:3(<module>) 1 0.000 0.000 102.580 102.580 pump_transfer.py:46(main) 1 0.000 0.000 0.006 0.006 pump_transfer.py:79(opt_parse) 1 0.000 0.000 0.003 0.003 pump_transfer.py:91(opt_parser) 1 0.000 0.000 0.003 0.003 pump_transfer.py:97(opt_parser_options) 1 0.000 0.000 0.029 0.029 random.py:100(seed) 1 0.002 0.002 0.032 0.032 random.py:40(<module>) 1 0.000 0.000 0.029 0.029 random.py:91(_ init _) 9 0.000 0.000 0.014 0.002 re.py:188(compile) 9 0.000 0.000 0.014 0.002 re.py:226(_compile) 3150/1 0.035 0.000 0.093 0.093 scanner.py:27(_scan_once) 1 0.008 0.008 0.008 0.008 socket.py:45(<module>) 1 0.000 0.000 0.002 0.002 socket.py:537(create_connection) 1 0.004 0.004 0.036 0.036 tempfile.py:18(<module>) 485 0.002 0.000 0.003 0.000 urllib.py:1199(<genexpr>) 485 0.003 0.000 0.004 0.000 urlparse.py:315(<genexpr>) 38 0.000 0.000 0.000 0.000 {_ctypes.sizeof} 1768 0.004 0.000 0.004 0.000 {chr} 27 0.000 0.000 0.000 0.000 {filter} 53 0.000 0.000 0.000 0.000 {getattr} 72 0.000 0.000 0.000 0.000 {globals} 37 0.000 0.000 0.000 0.000 {hasattr} 93 0.000 0.000 0.000 0.000 {id} 262 0.001 0.000 0.001 0.000 {isinstance} 883/868 0.002 0.000 0.002 0.000 {len} 10 0.001 0.000 0.001 0.000 {method 'acquire' of 'thread.lock' objects} 11 0.000 0.000 0.000 0.000 {method 'add' of 'set' objects} 3866 0.007 0.000 0.007 0.000 {method 'append' of 'list' objects} 2180 0.005 0.000 0.005 0.000 {method 'end' of '_sre.SRE_Match' objects} 11 0.000 0.000 0.000 0.000 {method 'extend' of 'list' objects} 205 0.000 0.000 0.000 0.000 {method 'find' of 'str' objects} 191 0.000 0.000 0.000 0.000 {method 'format' of 'str' objects} 224 0.000 0.000 0.000 0.000 {method 'get' of 'dict' objects} 7 0.000 0.000 0.000 0.000 {method 'getvalue' of 'cStringIO.StringO' objects} 2178 0.005 0.000 0.005 0.000 {method 'groups' of '_sre.SRE_Match' objects} 86 0.000 0.000 0.000 0.000 {method 'isalnum' of 'str' objects} 13 0.000 0.000 0.000 0.000 {method 'isdigit' of 'str' objects} 14 0.000 0.000 0.000 0.000 {method 'items' of 'dict' objects} 5 0.000 0.000 0.000 0.000 {method 'iteritems' of 'dict' objects} 6 0.000 0.000 0.000 0.000 {method 'iterkeys' of 'dict' objects} 18 0.000 0.000 0.000 0.000 {method 'join' of 'str' objects} 100 0.000 0.000 0.000 0.000 {method 'join' of 'unicode' objects} 4 0.000 0.000 0.000 0.000 {method 'ljust' of 'str' objects} 24 0.000 0.000 0.000 0.000 {method 'lower' of 'str' objects} 2180 0.014 0.000 0.014 0.000 {method 'match' of '_sre.SRE_Pattern' objects} 8 0.000 0.000 0.000 0.000 {method 'pop' of 'dict' objects} 8 0.000 0.000 0.000 0.000 {method 'pop' of 'list' objects} 206 0.012 0.000 0.012 0.000 {method 'recv' of '_socket.socket' objects} 6 0.000 0.000 0.000 0.000 {method 'release' of 'thread.lock' objects} 7 0.000 0.000 0.000 0.000 {method 'remove' of 'list' objects} 43 0.000 0.000 0.000 0.000 {method 'replace' of 'str' objects} 12 0.000 0.000 0.000 0.000 {method 'rjust' of 'str' objects} 17 0.000 0.000 0.000 0.000 {method 'seek' of 'cStringIO.StringO' objects} 32 0.000 0.000 0.000 0.000 {method 'setdefault' of 'dict' objects} 23 0.000 0.000 0.000 0.000 {method 'split' of 'str' objects} 106 0.000 0.000 0.000 0.000 {method 'startswith' of 'str' objects} 17 0.000 0.000 0.000 0.000 {method 'strip' of 'str' objects} 16 0.000 0.000 0.000 0.000 {method 'tell' of 'cStringIO.StringO' objects} 10 0.000 0.000 0.000 0.000 {method 'upper' of 'str' objects} 195 0.000 0.000 0.000 0.000 {method 'write' of 'cStringIO.StringO' objects} 7 0.004 0.001 0.004 0.001 {method 'write' of 'file' objects} 211 0.001 0.000 0.001 0.000 {min} 1 0.029 0.029 0.029 0.029 {nt.urandom} 39 0.000 0.000 0.000 0.000 {ord} 14 0.000 0.000 0.000 0.000 {range} 383 0.001 0.000 0.001 0.000 {setattr} 11 0.000 0.000 0.000 0.000 {thread.allocate_lock} 11 0.000 0.000 0.000 0.000 {thread.get_ident} 149 102.449 0.688 102.449 0.688 {time.sleep}
        Hide
        bcui Bin Cui (Inactive) added a comment -

        Looks like the majority time spent on time.sleep().

        Show
        bcui Bin Cui (Inactive) added a comment - Looks like the majority time spent on time.sleep().
        Hide
        bcui Bin Cui (Inactive) added a comment -

        We don't have new findings out of this observed issue. Close it unless we have something new found.

        Show
        bcui Bin Cui (Inactive) added a comment - We don't have new findings out of this observed issue. Close it unless we have something new found.

          People

          • Assignee:
            bcui Bin Cui (Inactive)
            Reporter:
            steve Steve Yen
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Agile

                Gerrit Reviews

                There are no open Gerrit changes