Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-5938

UI needs to deal with very slow docloader (alternatively, docloader is too slow)

    Details

      Description

      It works fine with old samples (gamesim-sample), but recent archives (beer-sample) contains much more data and usually fail due to timeout.

      Now it takes at least 40 seconds to load documents and create views.

      # Subject Project Status CR V
      For Gerrit Dashboard: &For+MB-5938=message:MB-5938

        Activity

        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        Hm. Users are not generally prepared to wait that much. In theory there should be no problem loading thousands of docs per second. I don't think beers data is mega docs.

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - Hm. Users are not generally prepared to wait that much. In theory there should be no problem loading thousands of docs per second. I don't think beers data is mega docs.
        Hide
        pavelpaulau Pavel Paulau added a comment -

        1. before loading these docs we have to create new bucket (if it doesn't exist).
        2. every document is a separate file (for beers we have more then 6000 files in archive). and current approach to read these files is pretty weird.
        3. we also create new ddocs/views and perform queries on new indices.

        if there is any problem with increasing timeout, I can fix point #2.

        Show
        pavelpaulau Pavel Paulau added a comment - 1. before loading these docs we have to create new bucket (if it doesn't exist). 2. every document is a separate file (for beers we have more then 6000 files in archive). and current approach to read these files is pretty weird. 3. we also create new ddocs/views and perform queries on new indices. if there is any problem with increasing timeout, I can fix point #2.
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        Increasing timeout is simple. But waiting 40 seconds is a bit too much for users to tolerate.

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - Increasing timeout is simple. But waiting 40 seconds is a bit too much for users to tolerate.
        Hide
        pavelpaulau Pavel Paulau added a comment -

        ok, fixing #2 doesn't help sometimes.

        python client have extremely low throughput (regardless environment). and this is behind cbdocloader.
        thus we need ~15-20 seconds to load these sample items.

        I understand that 40 seconds are too long. but nobody is going to fix/improve old sdk branch, which is only compatible with python 2.4.

        Show
        pavelpaulau Pavel Paulau added a comment - ok, fixing #2 doesn't help sometimes. python client have extremely low throughput (regardless environment). and this is behind cbdocloader. thus we need ~15-20 seconds to load these sample items. I understand that 40 seconds are too long. but nobody is going to fix/improve old sdk branch, which is only compatible with python 2.4.
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        Trying longer timeout. That sucks. We don't have good feedback for users.

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - Trying longer timeout. That sucks. We don't have good feedback for users.
        Hide
        dipti Dipti Borkar added a comment -

        Can we just make the load asynchronous and log problems?

        Show
        dipti Dipti Borkar added a comment - Can we just make the load asynchronous and log problems?
        Hide
        thuan Thuan Nguyen added a comment -

        Integrated in github-ns-server-2-0 #412 (See http://qa.hq.northscale.net/job/github-ns-server-2-0/412/)
        MB-5938: bumped up docloader timeout to 60 seconds (Revision f6d28497ddb2270962b8ba73284da66237120be8)

        Result = SUCCESS
        Aliaksey Artamonau :
        Files :

        • src/menelaus_web.erl
        Show
        thuan Thuan Nguyen added a comment - Integrated in github-ns-server-2-0 #412 (See http://qa.hq.northscale.net/job/github-ns-server-2-0/412/ ) MB-5938 : bumped up docloader timeout to 60 seconds (Revision f6d28497ddb2270962b8ba73284da66237120be8) Result = SUCCESS Aliaksey Artamonau : Files : src/menelaus_web.erl
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        Assigning to Pavel for UI-only attempt to fix

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - Assigning to Pavel for UI-only attempt to fix
        Hide
        pavel Pavel Blagodov added a comment -

        I had tried to POST on http://localhost:9001/sampleBuckets/install and after 2.8min of pending I got error 400 from server with the following response

        {"error":true,"type":"docloader_failed","reason":"There was an unexpected error."}

        In this case SAMPLES_LOADING_TIMEOUT was 12000000.

        Show
        pavel Pavel Blagodov added a comment - I had tried to POST on http://localhost:9001/sampleBuckets/install and after 2.8min of pending I got error 400 from server with the following response {"error":true,"type":"docloader_failed","reason":"There was an unexpected error."} In this case SAMPLES_LOADING_TIMEOUT was 12000000.
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        There's blocker level bug on docloader for that

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - There's blocker level bug on docloader for that
        Hide
        karan Karan Kumar (Inactive) added a comment -

        Also look at
        http://www.couchbase.com/issues/browse/MB-6274

        for other docloader improvements.

        Show
        karan Karan Kumar (Inactive) added a comment - Also look at http://www.couchbase.com/issues/browse/MB-6274 for other docloader improvements.
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        Not actually fixed.

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - Not actually fixed.
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        Sorry. Meant to assign to Pavel

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - Sorry. Meant to assign to Pavel
        Show
        pavel Pavel Blagodov added a comment - http://review.couchbase.org/20834
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        I've uploaded alternative commit to gerrit. Sample loading call will now return as soon as buckets are created, then data will be loaded in background and task will be reported via list of tasks (without progress report however) in right top corner.

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - I've uploaded alternative commit to gerrit. Sample loading call will now return as soon as buckets are created, then data will be loaded in background and task will be reported via list of tasks (without progress report however) in right top corner.
        Hide
        kzeller kzeller added a comment -

        Sample data will now be asynchronously loaded. Loading sample data will be
        executed as a background task then later reported in web console
        upon completion.

        Show
        kzeller kzeller added a comment - Sample data will now be asynchronously loaded. Loading sample data will be executed as a background task then later reported in web console upon completion.

          People

          • Assignee:
            alkondratenko Aleksey Kondratenko (Inactive)
            Reporter:
            pavelpaulau Pavel Paulau
          • Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes