Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-6473

stale=ok\update_after queries are failed with timeout error rebalance out 10->9 nodes

    XMLWordPrintable

Details

    • Bug
    • Resolution: Incomplete
    • Major
    • 2.0
    • None
    • None
    • Security Level: Public
    • None
    • 9 VMs, centOS, build #1645

    Description

      Timeout error appear when do query stale=ok\update_after during rebalance 10->9, index is built before rebalance start
      2 buckets: default and sasl bucket
      Data:
      loaded with mcsoda: default - 7M items, sasl-730K

      Views structure:
      1 ddoc, 2 views:
      v1 - function(doc)

      {emit(doc.key, doc.key_num);}

      v2 - function(doc, meta)

      {emit(meta.id, doc.key);}

      Error:
      Subset of nodes failed with the following error:
      [

      { "from": "http://10.3.121.20:8092/_view_merge/?limit=10", "reason": "timeout" }

      ,

      { "from": "http://10.3.121.13:8092/_view_merge/?limit=10", "reason": "timeout" }

      ,

      { "from": "http://10.3.121.15:8092/_view_merge/?limit=10", "reason": "timeout" }

      ]

      exact queries:
      iryna@ubuntu:~/couchbase/testrunner$ curl -v 'http://10.3.121.17:8092/default/_design/d1/_view/v2?stale=update_after&connection_timeout=60000&limit=10&skip=0'

      • About to connect() to 10.3.121.17 port 8092 (#0)
      • Trying 10.3.121.17...
      • connected
      • Connected to 10.3.121.17 (10.3.121.17) port 8092 (#0)
        > GET /default/_design/d1/_view/v2?stale=update_after&connection_timeout=60000&limit=10&skip=0 HTTP/1.1
        > User-Agent: curl/7.26.0
        > Host: 10.3.121.17:8092
        > Accept: /
        >
        < HTTP/1.1 200 OK
        < Transfer-Encoding: chunked
        < Server: MochiWeb/1.0 (Any of you quaids got a smint?)
        < Date: Wed, 29 Aug 2012 01:23:52 GMT
        < Content-Type: text/plain;charset=utf-8
        < Cache-Control: must-revalidate
        <
        {"total_rows":8649506,"rows":[ {"id":"0-0000046792785dbe","key":"0-0000046792785dbe","value":"0-0000046792785dbe"}

        ,

        {"id":"0-00000586045f623e","key":"0-00000586045f623e","value":"0-00000586045f623e"}

        ,

        {"id":"0-000009891526c0ad","key":"0-000009891526c0ad","value":"0-000009891526c0ad"}

        ,

        {"id":"0-00000ce845c00cbf","key":"0-00000ce845c00cbf","value":"0-00000ce845c00cbf"}

        ,

        {"id":"0-00000f7264c27ba6","key":"0-00000f7264c27ba6","value":"0-00000f7264c27ba6"}

        ,

        {"id":"0-0000104cd168386a","key":"0-0000104cd168386a","value":"0-0000104cd168386a"}

        ,

        {"id":"0-00002783bc5d1085","key":"0-00002783bc5d1085","value":"0-00002783bc5d1085"}

        ,

        {"id":"0-00004c3b8694276f","key":"0-00004c3b8694276f","value":"0-00004c3b8694276f"}

        ,

        {"id":"0-00004ea3c948ffe3","key":"0-00004ea3c948ffe3","value":"0-00004ea3c948ffe3"}

        ,

        {"id":"0-00004fb0041b47b8","key":"0-00004fb0041b47b8","value":"0-00004fb0041b47b8"}

        ],
        "errors":[

        {"from":"http://10.3.121.20:8092/_view_merge/?limit=10","reason":"timeout"}

        ,

        {"from":"http://10.3.121.15:8092/_view_merge/?limit=10","reason":"timeout"}

        ,

        {"from":"http://10.3.121.13:8092/_view_merge/?limit=10","reason":"timeout"}

        ]
        }

      • Connection #0 to host 10.3.121.17 left intact
      • Closing connection #0
        iryna@ubuntu:~/couchbase/testrunner$ curl -v 'http://10.3.121.13:8092/default/_design/d1/_view/v2?stale=update_after&connection_timeout=60000&limit=10&skip=0'
      • About to connect() to 10.3.121.13 port 8092 (#0)
      • Trying 10.3.121.13...
      • connected
      • Connected to 10.3.121.13 (10.3.121.13) port 8092 (#0)
        > GET /default/_design/d1/_view/v2?stale=update_after&connection_timeout=60000&limit=10&skip=0 HTTP/1.1
        > User-Agent: curl/7.26.0
        > Host: 10.3.121.13:8092
        > Accept: /
        >
        < HTTP/1.1 200 OK
        < Transfer-Encoding: chunked
        < Server: MochiWeb/1.0 (Any of you quaids got a smint?)
        < Date: Wed, 29 Aug 2012 01:34:35 GMT
        < Content-Type: text/plain;charset=utf-8
        < Cache-Control: must-revalidate
        <
        {"total_rows":10517139,"rows":[ {"id":"0-0000046792785dbe","key":"0-0000046792785dbe","value":"0-0000046792785dbe"}

        ,

        {"id":"0-00000586045f623e","key":"0-00000586045f623e","value":"0-00000586045f623e"}

        ,

        {"id":"0-00000ce845c00cbf","key":"0-00000ce845c00cbf","value":"0-00000ce845c00cbf"}

        ,

        {"id":"0-00000f7264c27ba6","key":"0-00000f7264c27ba6","value":"0-00000f7264c27ba6"}

        ,

        {"id":"0-0000104cd168386a","key":"0-0000104cd168386a","value":"0-0000104cd168386a"}

        ,

        {"id":"0-00002783bc5d1085","key":"0-00002783bc5d1085","value":"0-00002783bc5d1085"}

        ,

        {"id":"0-00002f9a9b2659cb","key":"0-00002f9a9b2659cb","value":"0-00002f9a9b2659cb"}

        ,

        {"id":"0-0000300773fe015f","key":"0-0000300773fe015f","value":"0-0000300773fe015f"}

        ,

        {"id":"0-00004c3b8694276f","key":"0-00004c3b8694276f","value":"0-00004c3b8694276f"}

        ,

        {"id":"0-00004ea3c948ffe3","key":"0-00004ea3c948ffe3","value":"0-00004ea3c948ffe3"}

        ],
        "errors":[

        {"from":"http://10.3.121.15:8092/_view_merge/?limit=10","reason":"timeout"}

        ]
        }

      • Connection #0 to host 10.3.121.13 left intact
      • Closing connection #0

      iryna@ubuntu:~/couchbase/testrunner$ curl -v 'http://10.3.121.13:8092/default/_design/d1/_view/v2?stale=ok&connection_timeout=60000&limit=10&skip=0'

      • About to connect() to 10.3.121.13 port 8092 (#0)
      • Trying 10.3.121.13...
      • connected
      • Connected to 10.3.121.13 (10.3.121.13) port 8092 (#0)
        > GET /default/_design/d1/_view/v2?stale=ok&connection_timeout=60000&limit=10&skip=0 HTTP/1.1
        > User-Agent: curl/7.26.0
        > Host: 10.3.121.13:8092
        > Accept: /
        >
        < HTTP/1.1 200 OK
        < Transfer-Encoding: chunked
        < Server: MochiWeb/1.0 (Any of you quaids got a smint?)
        < Date: Wed, 29 Aug 2012 01:39:09 GMT
        < Content-Type: text/plain;charset=utf-8
        < Cache-Control: must-revalidate
        <
        {"total_rows":11597601,"rows":[ {"id":"0-0000046792785dbe","key":"0-0000046792785dbe","value":"0-0000046792785dbe"}

        ,

        {"id":"0-00000586045f623e","key":"0-00000586045f623e","value":"0-00000586045f623e"}

        ,

        {"id":"0-00000ce845c00cbf","key":"0-00000ce845c00cbf","value":"0-00000ce845c00cbf"}

        ,

        {"id":"0-00000f7264c27ba6","key":"0-00000f7264c27ba6","value":"0-00000f7264c27ba6"}

        ,

        {"id":"0-0000104cd168386a","key":"0-0000104cd168386a","value":"0-0000104cd168386a"}

        ,

        {"id":"0-00002783bc5d1085","key":"0-00002783bc5d1085","value":"0-00002783bc5d1085"}

        ,

        {"id":"0-00002f9a9b2659cb","key":"0-00002f9a9b2659cb","value":"0-00002f9a9b2659cb"}

        ,

        {"id":"0-0000300773fe015f","key":"0-0000300773fe015f","value":"0-0000300773fe015f"}

        ,

        {"id":"0-000048290a305419","key":"0-000048290a305419","value":"0-000048290a305419"}

        ,

        {"id":"0-00004e32a448b483","key":"0-00004e32a448b483","value":"0-00004e32a448b483"}

        ],
        "errors":[

        {"from":"http://10.3.121.22:8092/_view_merge/?stale=ok&limit=10","reason":"timeout"}

        ]
        }

      • Connection #0 to host 10.3.121.13 left intact
      • Closing connection #0

      diags:
      https://s3.amazonaws.com/packages.couchbase/diag-logs/large_cluster_2_0/9nodes-1645-view-error-timeout-20120828.tgz

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            FilipeManana Filipe Manana (Inactive)
            iryna iryna
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty