Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-6633

querying for views during rebalancing through node's REST api before it has any active vbucket return ""error":"no_active_vbuckets"

    Details

      Description

      test
      -t viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_incrementaly,skip_rebalance=true

      steps to repro:
      1. load data
      2. start rebalance 1->2
      3. wait some time, (in my case rebalance is going on 15 mins) try to see documents or query view from node which is adding to cluster
      no_active_vbuckets error appears
      iryna@ubuntu:~/couchbase/testrunner$ curl -v 'http://10.3.121.105:8092/default/_design/test_view-df0f1d8/_view/test_view-df0f1d8?debug=true&start_key=%5B2008%2C7%2Cnull%5D&connection_timeout=60000'

      • About to connect() to 10.3.121.105 port 8092 (#0)
      • Trying 10.3.121.105...
      • connected
      • Connected to 10.3.121.105 (10.3.121.105) port 8092 (#0)
        > GET /default/_design/test_view-df0f1d8/_view/test_view-df0f1d8?debug=true&start_key=%5B2008%2C7%2Cnull%5D&connection_timeout=60000 HTTP/1.1
        > User-Agent: curl/7.26.0
        > Host: 10.3.121.105:8092
        > Accept: /
        >
        < HTTP/1.1 404 Object Not Found
        < Server: MochiWeb/1.0 (Any of you quaids got a smint?)
        < Date: Wed, 12 Sep 2012 22:07:23 GMT
        < Content-Type: text/plain;charset=utf-8
        < Content-Length: 106
        < Cache-Control: must-revalidate
        < {"error":"no_active_vbuckets","reason":"Cannot execute view query since the node has no active vbuckets"}
      • Connection #0 to host 10.3.121.105 left intact
      • Closing connection #0
      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        iryna iryna created issue -
        Hide
        farshid Farshid Ghods (Inactive) added a comment -

        looks like a blocker to me , is this regression on 1717 ? have you been able to reproduce ths on earlier builds ?

        Show
        farshid Farshid Ghods (Inactive) added a comment - looks like a blocker to me , is this regression on 1717 ? have you been able to reproduce ths on earlier builds ?
        Hide
        iryna iryna added a comment -

        first seen on 1715

        Show
        iryna iryna added a comment - first seen on 1715
        iryna iryna made changes -
        Field Original Value New Value
        Priority Major [ 3 ] Blocker [ 1 ]
        Hide
        farshid Farshid Ghods (Inactive) added a comment -

        ok this could be related to ep-engine changes?

        what do you see in diags ?
        can you do more investigations

        Show
        farshid Farshid Ghods (Inactive) added a comment - ok this could be related to ep-engine changes? what do you see in diags ? can you do more investigations
        farshid Farshid Ghods (Inactive) made changes -
        Assignee Aleksey Kondratenko [ alkondratenko ] Iryna Mironava [ iryna ]
        iryna iryna made changes -
        Summary unable to access documents and query views from rebalancing in from 1 to 2 nodes for node that is adding to cluster: unable to access documents and query views from rebalancing in from 1 to 2 nodes
        Hide
        iryna iryna added a comment -

        it doesn't seem to be an ep-engine problem, we can query views from nodes which was in cluster successfully, the error appears only if we query node that we try to add. In logs i don't see something wrong, it is building replicas.

        Show
        iryna iryna added a comment - it doesn't seem to be an ep-engine problem, we can query views from nodes which was in cluster successfully, the error appears only if we query node that we try to add. In logs i don't see something wrong, it is building replicas.
        Hide
        farshid Farshid Ghods (Inactive) added a comment -

        Aliaksey,

        I dont think this is a regression but the use cases is too simple ( doing view queries from a node that is being rebalanced in but its still doing replica building and does not have an active vbucket )

        Show
        farshid Farshid Ghods (Inactive) added a comment - Aliaksey, I dont think this is a regression but the use cases is too simple ( doing view queries from a node that is being rebalanced in but its still doing replica building and does not have an active vbucket )
        farshid Farshid Ghods (Inactive) made changes -
        Component/s ns_server [ 10019 ]
        Component/s view-engine [ 10060 ]
        farshid Farshid Ghods (Inactive) made changes -
        Assignee Iryna Mironava [ iryna ] Aleksey Kondratenko [ alkondratenko ]
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        I need 100% clarity here. Does that happen after rebalance or during rebalance.

        I'm asking because sometimes we need quite a lot of time to start moving any active vbuckets into new node. So if it's not after, then it's simple duplicate of MB-6440.

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - I need 100% clarity here. Does that happen after rebalance or during rebalance. I'm asking because sometimes we need quite a lot of time to start moving any active vbuckets into new node. So if it's not after, then it's simple duplicate of MB-6440 .
        alkondratenko Aleksey Kondratenko (Inactive) made changes -
        Assignee Aleksey Kondratenko [ alkondratenko ] Iryna Mironava [ iryna ]
        Hide
        farshid Farshid Ghods (Inactive) added a comment -

        MB-6440 says the error only appears for a few second after rebalancing this bug says it happens even after 10 minutes post rebalancing

        Show
        farshid Farshid Ghods (Inactive) added a comment - MB-6440 says the error only appears for a few second after rebalancing this bug says it happens even after 10 minutes post rebalancing
        Hide
        iryna iryna added a comment -

        it happens during rebalance, but for much longer time window, 10-15 minutes i cannot see documents and query views. Is it a duplicate?

        Show
        iryna iryna added a comment - it happens during rebalance, but for much longer time window, 10-15 minutes i cannot see documents and query views. Is it a duplicate?
        iryna iryna made changes -
        Assignee Iryna Mironava [ iryna ] Aleksey Kondratenko [ alkondratenko ]
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        Yes it's duplicate.

        It is reasonably simple to fix, but it may have some quite bad effects on clients.

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - Yes it's duplicate. It is reasonably simple to fix, but it may have some quite bad effects on clients.
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        BTW this is not AFAIK a regression. Do you want me to fix this ASAP?

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - BTW this is not AFAIK a regression. Do you want me to fix this ASAP?
        Hide
        farshid Farshid Ghods (Inactive) added a comment -

        moving this to beta refresh so once beta is shipped we will merge it

        Show
        farshid Farshid Ghods (Inactive) added a comment - moving this to beta refresh so once beta is shipped we will merge it
        farshid Farshid Ghods (Inactive) made changes -
        Fix Version/s 2.0-beta-refresh [ 10385 ]
        Fix Version/s 2.0-beta [ 10113 ]
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        Some fix is coming. But now I understand there's another cause of this. I think enabling consistent views will fix other and potentially more prolonged it time cause.

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - Some fix is coming. But now I understand there's another cause of this. I think enabling consistent views will fix other and potentially more prolonged it time cause.
        farshid Farshid Ghods (Inactive) made changes -
        Summary for node that is adding to cluster: unable to access documents and query views from rebalancing in from 1 to 2 nodes querying for views during rebalancing through node's REST api before it has any active vbucket return ""error":"no_active_vbuckets"
        Labels regression 2.0-beta-release-notes regression
        Hide
        kzeller kzeller added a comment -

        Add to beta release notes as : If a cluster is being rebalanced, and a node does not yet have any active
        vBuckets, you will receive the error
        <literal>"error":"no_active_vbuckets"</literal> when you query a view
        via the REST API.

        Show
        kzeller kzeller added a comment - Add to beta release notes as : If a cluster is being rebalanced, and a node does not yet have any active vBuckets, you will receive the error <literal>"error":"no_active_vbuckets"</literal> when you query a view via the REST API.
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        merged fix

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - merged fix
        alkondratenko Aleksey Kondratenko (Inactive) made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        farshid Farshid Ghods (Inactive) made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            alkondratenko Aleksey Kondratenko (Inactive)
            Reporter:
            iryna iryna
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes