Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-5938

UI needs to deal with very slow docloader (alternatively, docloader is too slow)

    Details

      Description

      It works fine with old samples (gamesim-sample), but recent archives (beer-sample) contains much more data and usually fail due to timeout.

      Now it takes at least 40 seconds to load documents and create views.

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        pavelpaulau Pavel Paulau created issue -
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        Hm. Users are not generally prepared to wait that much. In theory there should be no problem loading thousands of docs per second. I don't think beers data is mega docs.

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - Hm. Users are not generally prepared to wait that much. In theory there should be no problem loading thousands of docs per second. I don't think beers data is mega docs.
        Hide
        pavelpaulau Pavel Paulau added a comment -

        1. before loading these docs we have to create new bucket (if it doesn't exist).
        2. every document is a separate file (for beers we have more then 6000 files in archive). and current approach to read these files is pretty weird.
        3. we also create new ddocs/views and perform queries on new indices.

        if there is any problem with increasing timeout, I can fix point #2.

        Show
        pavelpaulau Pavel Paulau added a comment - 1. before loading these docs we have to create new bucket (if it doesn't exist). 2. every document is a separate file (for beers we have more then 6000 files in archive). and current approach to read these files is pretty weird. 3. we also create new ddocs/views and perform queries on new indices. if there is any problem with increasing timeout, I can fix point #2.
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        Increasing timeout is simple. But waiting 40 seconds is a bit too much for users to tolerate.

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - Increasing timeout is simple. But waiting 40 seconds is a bit too much for users to tolerate.
        Hide
        pavelpaulau Pavel Paulau added a comment -

        ok, fixing #2 doesn't help sometimes.

        python client have extremely low throughput (regardless environment). and this is behind cbdocloader.
        thus we need ~15-20 seconds to load these sample items.

        I understand that 40 seconds are too long. but nobody is going to fix/improve old sdk branch, which is only compatible with python 2.4.

        Show
        pavelpaulau Pavel Paulau added a comment - ok, fixing #2 doesn't help sometimes. python client have extremely low throughput (regardless environment). and this is behind cbdocloader. thus we need ~15-20 seconds to load these sample items. I understand that 40 seconds are too long. but nobody is going to fix/improve old sdk branch, which is only compatible with python 2.4.
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        Trying longer timeout. That sucks. We don't have good feedback for users.

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - Trying longer timeout. That sucks. We don't have good feedback for users.
        alkondratenko Aleksey Kondratenko (Inactive) made changes -
        Field Original Value New Value
        Summary Timeout for loading sample buckets is too low UI needs to deal with very slow docloader (alternatively, docloader is too slow) [was: Timeout for loading sample buckets is too low]
        dipti Dipti Borkar made changes -
        Fix Version/s recent-builds-2.0 [ 10349 ]
        Sprint Status Current Sprint
        Sprint Priority 5
        farshid Farshid Ghods (Inactive) made changes -
        Fix Version/s 2.0-beta [ 10113 ]
        Affects Version/s 2.0-beta [ 10113 ]
        Hide
        dipti Dipti Borkar added a comment -

        Can we just make the load asynchronous and log problems?

        Show
        dipti Dipti Borkar added a comment - Can we just make the load asynchronous and log problems?
        Hide
        thuan Thuan Nguyen added a comment -

        Integrated in github-ns-server-2-0 #412 (See http://qa.hq.northscale.net/job/github-ns-server-2-0/412/)
        MB-5938: bumped up docloader timeout to 60 seconds (Revision f6d28497ddb2270962b8ba73284da66237120be8)

        Result = SUCCESS
        Aliaksey Artamonau :
        Files :

        • src/menelaus_web.erl
        Show
        thuan Thuan Nguyen added a comment - Integrated in github-ns-server-2-0 #412 (See http://qa.hq.northscale.net/job/github-ns-server-2-0/412/ ) MB-5938 : bumped up docloader timeout to 60 seconds (Revision f6d28497ddb2270962b8ba73284da66237120be8) Result = SUCCESS Aliaksey Artamonau : Files : src/menelaus_web.erl
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        Assigning to Pavel for UI-only attempt to fix

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - Assigning to Pavel for UI-only attempt to fix
        alkondratenko Aleksey Kondratenko (Inactive) made changes -
        Assignee Aleksey Kondratenko [ alkondratenko ] Pavel Blagodov [ pavel ]
        pavel Pavel Blagodov made changes -
        Status Open [ 1 ] In Progress [ 3 ]
        Hide
        pavel Pavel Blagodov added a comment -

        I had tried to POST on http://localhost:9001/sampleBuckets/install and after 2.8min of pending I got error 400 from server with the following response

        {"error":true,"type":"docloader_failed","reason":"There was an unexpected error."}

        In this case SAMPLES_LOADING_TIMEOUT was 12000000.

        Show
        pavel Pavel Blagodov added a comment - I had tried to POST on http://localhost:9001/sampleBuckets/install and after 2.8min of pending I got error 400 from server with the following response {"error":true,"type":"docloader_failed","reason":"There was an unexpected error."} In this case SAMPLES_LOADING_TIMEOUT was 12000000.
        pavel Pavel Blagodov made changes -
        Assignee Pavel Blagodov [ pavel ] Aleksey Kondratenko [ alkondratenko ]
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        There's blocker level bug on docloader for that

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - There's blocker level bug on docloader for that
        Hide
        karan Karan Kumar (Inactive) added a comment -

        Also look at
        http://www.couchbase.com/issues/browse/MB-6274

        for other docloader improvements.

        Show
        karan Karan Kumar (Inactive) added a comment - Also look at http://www.couchbase.com/issues/browse/MB-6274 for other docloader improvements.
        peter peter made changes -
        Status In Progress [ 3 ] Closed [ 6 ]
        Resolution Fixed [ 1 ]
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        Not actually fixed.

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - Not actually fixed.
        alkondratenko Aleksey Kondratenko (Inactive) made changes -
        Resolution Fixed [ 1 ]
        Status Closed [ 6 ] Reopened [ 4 ]
        alkondratenko Aleksey Kondratenko (Inactive) made changes -
        Assignee Aleksey Kondratenko [ alkondratenko ] John (alternate) Zablocki [ jzablocki ]
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        Sorry. Meant to assign to Pavel

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - Sorry. Meant to assign to Pavel
        alkondratenko Aleksey Kondratenko (Inactive) made changes -
        Assignee John (alternate) Zablocki [ jzablocki ] Pavel Blagodov [ pavel ]
        pavel Pavel Blagodov made changes -
        Status Reopened [ 4 ] In Progress [ 3 ]
        dipti Dipti Borkar made changes -
        Fix Version/s 2.0-beta-refresh [ 10385 ]
        Fix Version/s 2.0-beta [ 10113 ]
        dipti Dipti Borkar made changes -
        Priority Major [ 3 ] Blocker [ 1 ]
        Sprint Priority 5 2
        farshid Farshid Ghods (Inactive) made changes -
        Labels 2.0-beta-release-notes
        dipti Dipti Borkar made changes -
        Summary UI needs to deal with very slow docloader (alternatively, docloader is too slow) [was: Timeout for loading sample buckets is too low] UI needs to deal with very slow docloader (alternatively, docloader is too slow)
        Show
        pavel Pavel Blagodov added a comment - http://review.couchbase.org/20834
        pavel Pavel Blagodov made changes -
        Status In Progress [ 3 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        I've uploaded alternative commit to gerrit. Sample loading call will now return as soon as buckets are created, then data will be loaded in background and task will be reported via list of tasks (without progress report however) in right top corner.

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - I've uploaded alternative commit to gerrit. Sample loading call will now return as soon as buckets are created, then data will be loaded in background and task will be reported via list of tasks (without progress report however) in right top corner.
        alkondratenko Aleksey Kondratenko (Inactive) made changes -
        Assignee Pavel Blagodov [ pavel ] Aleksey Kondratenko [ alkondratenko ]
        Hide
        kzeller kzeller added a comment -

        Sample data will now be asynchronously loaded. Loading sample data will be
        executed as a background task then later reported in web console
        upon completion.

        Show
        kzeller kzeller added a comment - Sample data will now be asynchronously loaded. Loading sample data will be executed as a background task then later reported in web console upon completion.
        maria Maria McDuff (Inactive) made changes -
        Fix Version/s 2.0 [ 10114 ]
        Fix Version/s recent-builds-2.0 [ 10349 ]
        pavelpaulau Pavel Paulau made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            alkondratenko Aleksey Kondratenko (Inactive)
            Reporter:
            pavelpaulau Pavel Paulau
          • Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes