Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 1.2.0
    • Fix Version/s: 1.2.1
    • Component/s: None
    • Labels:
      None
    • Environment:
      Cluster 1974, SDK Versions (master at time of writing)

      Description

      [INFO 32.82 cbsdk.scenario rebalance.py:207] Sleeping 30 seconds for ramp-up
      [ACTION 62.84 cbsdk.scenario rebalance.py:210] Starting rebalance action
      [SDKD(INFO) 65.22 cbsdk.sdkd.remote remote.py:263] [Couchbase.CouchbaseViewHandler|Warn] View Couchbase.CouchbaseViewHandler was mapped to a dead node, failing.
      [SDKD(INFO) 65.22 cbsdk.sdkd.remote remote.py:263] [Sdkd.ViewQuery|Error] ... System.InvalidOperationException Operation is not valid due to the current state of the object. at Couchbase.CouchbaseViewHandler.GetResponse(IDictionary`2 viewParams) in c:\ncbc\src\Couchbase\CouchbaseViewHandler.cs:line 131
      [SDKD(INFO) 65.22 cbsdk.sdkd.remote remote.py:263] at Couchbase.CouchbaseViewHandler.<TransformResults>d__0`1.MoveNext() in c:\ncbc\src\Couchbase\CouchbaseViewHandler.cs:line 38
      [SDKD(INFO) 65.22 cbsdk.sdkd.remote remote.py:263] at Sdkd.Views.ViewCommand.Run() in c:\Users\Administrator\Documents\GitHub\sdkd-net\src\Sdkd\Views\ViewCommand.cs:line 219
      [SDKD(INFO) 65.91 cbsdk.sdkd.remote remote.py:263] [Couchbase.CouchbaseViewHandler|Warn] View Couchbase.CouchbaseViewHandler was mapped to a dead node, failing.
      [SDKD(INFO) 65.91 cbsdk.sdkd.remote remote.py:263] [Couchbase.CouchbaseViewHandler|Warn] View Couchbase.CouchbaseViewHandler was mapped to a dead node, failing.
      [SDKD(INFO) 65.91 cbsdk.sdkd.remote remote.py:263] [Couchbase.CouchbaseViewHandler|Warn] View Couchbase.CouchbaseViewHandler was mapped to a dead node, failing.
      [SDKD(INFO) 65.91 cbsdk.sdkd.remote remote.py:263] [Couchbase.CouchbaseViewHandler|Warn] View Couchbase.CouchbaseViewHandler was mapped to a dead node, failing.
      [SDKD(INFO) 65.92 cbsdk.sdkd.remote remote.py:263] [Couchbase.CouchbaseViewHandler|Warn] View Couchbase.CouchbaseViewHandler was mapped to a dead node, failing.
      [SDKD(INFO) 65.92 cbsdk.sdkd.remote remote.py:263] [Couchbase.CouchbaseViewHandler|Warn] View Couchbase.CouchbaseViewHandler was mapped to a dead node, failing.
      [SDKD(INFO) 65.92 cbsdk.sdkd.remote remote.py:263] [Couchbase.CouchbaseViewHandler|Warn] View Couchbase.CouchbaseViewHandler was mapped to a dead node, failing.
      [SDKD(INFO) 65.92 cbsdk.sdkd.remote remote.py:263] [Couchbase.CouchbaseViewHandler|Warn] View Couchbase.CouchbaseViewHandler was mapped to a dead node, failing.
      [SDKD(INFO) 65.92 cbsdk.sdkd.remote remote.py:263] [Couchbase.CouchbaseViewHandler|Warn] View Couchbase.CouchbaseViewHandler was mapped to a dead node, failing.
      [SDKD(INFO) 65.92 cbsdk.sdkd.remote remote.py:263] [Couchbase.CouchbaseViewHandler|Warn] View Couchbase.CouchbaseViewHandler was mapped to a dead node, failing.
      [SDKD(INFO) 65.92 cbsdk.sdkd.remote remote.py:263] [Couchbase.CouchbaseViewHandler|Warn] View Couchbase.CouchbaseViewHandler was mapped to a dead node, failing.
      [SDKD(INFO) 65.92 cbsdk.sdkd.remote remote.py:263] [Couchbase.CouchbaseViewHandler|Warn] View Couchbase.CouchbaseViewHandler was mapped to a dead node, failing.
      [SDKD(INFO) 65.92 cbsdk.sdkd.remote remote.py:263] [Couchbase.CouchbaseViewHandler|Warn] View Couchbase.CouchbaseViewHandler was mapped to a dead node, failing.
      [SDKD(INFO) 65.92 cbsdk.sdkd.remote remote.py:263] [Couchbase.CouchbaseViewHandler|Warn] View Couchbase.CouchbaseViewHandler was mapped to a dead node, failing.
      [SDKD(INFO) 65.92 cbsdk.sdkd.remote remote.py:263] [Couchbase.CouchbaseViewHandler|Warn] View Couchbase.CouchbaseViewHandler was mapped to a dead node, failing.
      [SDKD(INFO) 65.92 cbsdk.sdkd.remote remote.py:263] [Couchbase.CouchbaseViewHandler|Warn] View Couchbase.CouchbaseViewHandler was mapped to a dead node, failing.
      [SDKD(INFO) 65.92 cbsdk.sdkd.remote remote.py:263] [Couchbase.CouchbaseViewHandler|Warn] View Couchbase.CouchbaseViewHandler was mapped to a dead node, failing.
      [SDKD(INFO) 65.92 cbsdk.sdkd.remote remote.py:263] [Couchbase.CouchbaseViewHandler|Warn] View Couchbase.CouchbaseViewHandler was mapped to a dead node, failing.
      [SDKD(INFO) 65.92 cbsdk.sdkd.remote remote.py:263] [Couchbase.CouchbaseViewHandler|Warn] View Couchbase.CouchbaseViewHandler was mapped to a dead node, failing.
      [SDKD(INFO) 65.92 cbsdk.sdkd.remote remote.py:263] [Couchbase.CouchbaseViewHandler|Warn] View Couchbase.CouchbaseViewHandler was mapped to a dead node, failing.
      [SDKD(INFO) 65.92 cbsdk.sdkd.remote remote.py:263] [Couchbase.CouchbaseViewHandler|Warn] View Couchbase.CouchbaseViewHandler was mapped to a dead node, failing.
      [SDKD(INFO) 66.24 cbsdk.sdkd.remote remote.py:263] [Couchbase.CouchbaseViewHandler|Warn] View Couchbase.CouchbaseViewHandler was mapped to a dead node, failing.
      [SDKD(INFO) 66.24 cbsdk.sdkd.remote remote.py:263] [Couchbase.CouchbaseViewHandler|Warn] View Couchbase.CouchbaseViewHandler was mapped to a dead node, failing.
      [SDKD(INFO) 66.24 cbsdk.sdkd.remote remote.py:263] [Couchbase.CouchbaseViewHandler|Warn] View Couchbase.CouchbaseViewHandler was mapped to a dead node, failing.
      [SDKD(INFO) 66.24 cbsdk.sdkd.remote remote.py:263] [Couchbase.CouchbaseViewHandler|Warn] View Couchbase.CouchbaseViewHandler was mapped to a dead node, failing.
      [SDKD(INFO) 66.24 cbsdk.sdkd.remote remote.py:263] [Couchbase.CouchbaseViewHandler|Warn] View Couchbase.CouchbaseViewHandler was mapped to a dead node, failing.
      [SDKD(INFO) 66.24 cbsdk.sdkd.remote remote.py:263] [Couchbase.CouchbaseViewHandler|Warn] View Couchbase.CouchbaseViewHandler was mapped to a dead node, failing.
      [SDKD(INFO) 66.24 cbsdk.sdkd.remote remote.py:263] [Couchbase.CouchbaseViewHandler|Warn] View Couchbase.CouchbaseViewHandler was mapped to a dead node, failing.
      [SDKD(INFO) 66.24 cbsdk.sdkd.remote remote.py:263] [Couchbase.CouchbaseViewHandler|Warn] View Couchbase.CouchbaseViewHandler was mapped to a dead node, failing.
      [SDKD(INFO) 66.24 cbsdk.sdkd.remote remote.py:263] [Couchbase.CouchbaseViewHandler|Warn] View Couchbase.CouchbaseViewHandler was mapped to a dead node, failing.
      [SDKD(INFO) 66.82 cbsdk.sdkd.remote remote.py:263] [Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl|Error] Could not init pool. System.NullReferenceException Object reference not set to an instance of an object. at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.CreateSocket() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 279
      [SDKD(INFO) 66.82 cbsdk.sdkd.remote remote.py:263] at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.InitPool() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 257
      [SDKD(INFO) 66.91 cbsdk.sdkd.remote remote.py:263] [Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl|Error] Could not init pool. System.NullReferenceException Object reference not set to an instance of an object. at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.CreateSocket() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 279
      [SDKD(INFO) 66.91 cbsdk.sdkd.remote remote.py:263] at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.InitPool() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 257
      [SDKD(INFO) 67.53 cbsdk.sdkd.remote remote.py:263] [Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl|Error] Could not init pool. System.NullReferenceException Object reference not set to an instance of an object. at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.CreateSocket() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 279
      [SDKD(INFO) 67.53 cbsdk.sdkd.remote remote.py:263] at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.InitPool() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 257
      [SDKD(INFO) 67.76 cbsdk.sdkd.remote remote.py:263] [Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl|Error] Could not init pool. System.NullReferenceException Object reference not set to an instance of an object. at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.CreateSocket() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 279
      [SDKD(INFO) 67.76 cbsdk.sdkd.remote remote.py:263] at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.InitPool() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 257
      [SDKD(INFO) 67.76 cbsdk.sdkd.remote remote.py:263] [Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl|Error] Could not init pool. System.NullReferenceException Object reference not set to an instance of an object. at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.CreateSocket() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 279
      [SDKD(INFO) 67.76 cbsdk.sdkd.remote remote.py:263] at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.InitPool() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 257
      [SDKD(INFO) 68.06 cbsdk.sdkd.remote remote.py:263] [Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl|Error] Could not init pool. System.NullReferenceException Object reference not set to an instance of an object. at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.CreateSocket() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 279
      [SDKD(INFO) 68.06 cbsdk.sdkd.remote remote.py:263] at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.InitPool() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 257
      [SDKD(INFO) 70.01 cbsdk.sdkd.remote remote.py:263] [Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl|Error] Could not init pool. System.NullReferenceException Object reference not set to an instance of an object. at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.CreateSocket() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 279
      [SDKD(INFO) 70.01 cbsdk.sdkd.remote remote.py:263] at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.InitPool() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 257
      [SDKD(INFO) 70.01 cbsdk.sdkd.remote remote.py:263] [Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl|Error] Could not init pool. System.NullReferenceException Object reference not set to an instance of an object. at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.CreateSocket() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 279
      [SDKD(INFO) 70.01 cbsdk.sdkd.remote remote.py:263] at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.InitPool() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 257
      [SDKD(INFO) 70.02 cbsdk.sdkd.remote remote.py:263] [Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl|Error] Could not init pool. System.NullReferenceException Object reference not set to an instance of an object. at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.CreateSocket() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 279
      [SDKD(INFO) 70.02 cbsdk.sdkd.remote remote.py:263] at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.InitPool() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 257
      [SDKD(INFO) 70.07 cbsdk.sdkd.remote remote.py:263] [Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl|Error] Could not init pool. System.NullReferenceException Object reference not set to an instance of an object. at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.CreateSocket() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 279
      [SDKD(INFO) 70.07 cbsdk.sdkd.remote remote.py:263] at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.InitPool() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 257
      [SDKD(INFO) 70.07 cbsdk.sdkd.remote remote.py:263] [Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl|Error] Could not init pool. System.NullReferenceException Object reference not set to an instance of an object. at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.CreateSocket() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 279
      [SDKD(INFO) 70.07 cbsdk.sdkd.remote remote.py:263] at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.InitPool() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 257
      [SDKD(INFO) 70.20 cbsdk.sdkd.remote remote.py:263] [Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl|Error] Could not init pool. System.NullReferenceException Object reference not set to an instance of an object. at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.CreateSocket() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 279
      [SDKD(INFO) 70.20 cbsdk.sdkd.remote remote.py:263] at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.InitPool() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 257
      [SDKD(INFO) 70.20 cbsdk.sdkd.remote remote.py:263] [Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl|Error] Could not init pool. System.NullReferenceException Object reference not set to an instance of an object. at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.CreateSocket() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 279
      [SDKD(INFO) 70.20 cbsdk.sdkd.remote remote.py:263] at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.InitPool() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 257
      [SDKD(INFO) 70.22 cbsdk.sdkd.remote remote.py:263] [Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl|Error] Could not init pool. System.NullReferenceException Object reference not set to an instance of an object. at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.CreateSocket() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 279
      [SDKD(INFO) 70.22 cbsdk.sdkd.remote remote.py:263] at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.InitPool() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 257
      [SDKD(INFO) 70.22 cbsdk.sdkd.remote remote.py:263] [Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl|Error] Could not init pool. System.NullReferenceException Object reference not set to an instance of an object. at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.CreateSocket() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 279
      [SDKD(INFO) 70.22 cbsdk.sdkd.remote remote.py:263] at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.InitPool() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 257
      [SDKD(INFO) 70.22 cbsdk.sdkd.remote remote.py:263] [Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl|Error] Could not init pool. System.NullReferenceException Object reference not set to an instance of an object. at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.CreateSocket() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 279
      [SDKD(INFO) 70.22 cbsdk.sdkd.remote remote.py:263] at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.InitPool() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 257
      [SDKD(INFO) 70.23 cbsdk.sdkd.remote remote.py:263] [Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl|Error] Could not init pool. System.NullReferenceException Object reference not set to an instance of an object. at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.CreateSocket() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 279
      [SDKD(INFO) 70.23 cbsdk.sdkd.remote remote.py:263] at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.InitPool() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 257
      [SDKD(INFO) 70.28 cbsdk.sdkd.remote remote.py:263] [Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl|Error] Could not init pool. System.NullReferenceException Object reference not set to an instance of an object. at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.CreateSocket() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 279
      [SDKD(INFO) 70.28 cbsdk.sdkd.remote remote.py:263] at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.InitPool() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 257
      [SDKD(INFO) 70.82 cbsdk.sdkd.remote remote.py:263] [Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl|Error] Could not init pool. System.NullReferenceException Object reference not set to an instance of an object. at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.CreateSocket() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 279
      [SDKD(INFO) 70.82 cbsdk.sdkd.remote remote.py:263] at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.InitPool() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 257
      [SDKD(INFO) 70.84 cbsdk.sdkd.remote remote.py:263] [Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl|Error] Could not init pool. System.NullReferenceException Object reference not set to an instance of an object. at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.CreateSocket() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 279
      [SDKD(INFO) 70.84 cbsdk.sdkd.remote remote.py:263] at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.InitPool() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 257
      [SDKD(INFO) 70.87 cbsdk.sdkd.remote remote.py:263] [Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl|Error] Could not init pool. System.NullReferenceException Object reference not set to an instance of an object. at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.CreateSocket() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 279
      [SDKD(INFO) 70.87 cbsdk.sdkd.remote remote.py:263] at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.InitPool() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 257
      [SDKD(INFO) 71.16 cbsdk.sdkd.remote remote.py:263] [Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl|Error] Could not init pool. System.NullReferenceException Object reference not set to an instance of an object. at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.CreateSocket() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 279
      [SDKD(INFO) 71.16 cbsdk.sdkd.remote remote.py:263] at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.InitPool() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 257
      [SDKD(INFO) 72.13 cbsdk.sdkd.remote remote.py:263] [Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl|Error] Could not init pool. System.NullReferenceException Object reference not set to an instance of an object. at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.CreateSocket() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 279
      [SDKD(INFO) 72.13 cbsdk.sdkd.remote remote.py:263] at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.InitPool() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 257
      [SDKD(INFO) 72.32 cbsdk.sdkd.remote remote.py:263] [Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl|Error] Could not init pool. System.NullReferenceException Object reference not set to an instance of an object. at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.CreateSocket() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 279
      [SDKD(INFO) 72.32 cbsdk.sdkd.remote remote.py:263] at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.InitPool() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 257
      [SDKD(INFO) 72.34 cbsdk.sdkd.remote remote.py:263] [Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl|Error] Could not init pool. System.NullReferenceException Object reference not set to an instance of an object. at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.CreateSocket() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 279
      [SDKD(INFO) 72.34 cbsdk.sdkd.remote remote.py:263] at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.InitPool() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 257
      [SDKD(INFO) 72.35 cbsdk.sdkd.remote remote.py:263] [Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl|Error] Could not init pool. System.NullReferenceException Object reference not set to an instance of an object. at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.CreateSocket() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 279
      [SDKD(INFO) 72.35 cbsdk.sdkd.remote remote.py:263] at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.InitPool() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 257
      [SDKD(INFO) 72.74 cbsdk.sdkd.remote remote.py:263] [Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl|Error] Could not init pool. System.NullReferenceException Object reference not set to an instance of an object. at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.CreateSocket() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 279
      [SDKD(INFO) 72.74 cbsdk.sdkd.remote remote.py:263] at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.InitPool() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 257
      [SDKD(INFO) 73.44 cbsdk.sdkd.remote remote.py:263] [Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl|Error] Could not init pool. System.NullReferenceException Object reference not set to an instance of an object. at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.CreateSocket() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 279
      [SDKD(INFO) 73.44 cbsdk.sdkd.remote remote.py:263] at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.InitPool() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 257
      [SDKD(INFO) 73.46 cbsdk.sdkd.remote remote.py:263] [Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl|Error] Could not init pool. System.NullReferenceException Object reference not set to an instance of an object. at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.CreateSocket() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 279
      [SDKD(INFO) 73.46 cbsdk.sdkd.remote remote.py:263] at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.InitPool() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 257
      [SDKD(INFO) 73.97 cbsdk.sdkd.remote remote.py:263] [Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl|Error] Could not init pool. System.NullReferenceException Object reference not set to an instance of an object. at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.CreateSocket() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 279
      [SDKD(INFO) 73.97 cbsdk.sdkd.remote remote.py:263] at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.InitPool() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 257
      [SDKD(INFO) 74.63 cbsdk.sdkd.remote remote.py:263] [Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl|Error] Could not init pool. System.NullReferenceException Object reference not set to an instance of an object. at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.CreateSocket() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 279
      [SDKD(INFO) 74.63 cbsdk.sdkd.remote remote.py:263] at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.InitPool() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 257
      [SDKD(INFO) 74.65 cbsdk.sdkd.remote remote.py:263] [Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl|Error] Could not init pool. System.NullReferenceException Object reference not set to an instance of an object. at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.CreateSocket() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 279
      [SDKD(INFO) 74.65 cbsdk.sdkd.remote remote.py:263] at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.InitPool() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 257
      [SDKD(INFO) 74.65 cbsdk.sdkd.remote remote.py:263] [Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl|Error] Could not init pool. System.NullReferenceException Object reference not set to an instance of an object. at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.CreateSocket() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 279
      [SDKD(INFO) 74.65 cbsdk.sdkd.remote remote.py:263] at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.InitPool() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 257
      [SDKD(INFO) 74.65 cbsdk.sdkd.remote remote.py:263] [Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl|Error] Could not init pool. System.NullReferenceException Object reference not set to an instance of an object. at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.CreateSocket() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 279
      [SDKD(INFO) 74.65 cbsdk.sdkd.remote remote.py:263] at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.InitPool() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 257
      [SDKD(INFO) 74.65 cbsdk.sdkd.remote remote.py:263] [Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl|Error] Could not init pool. System.NullReferenceException Object reference not set to an instance of an object. at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.CreateSocket() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 279
      [SDKD(INFO) 74.65 cbsdk.sdkd.remote remote.py:263] at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.InitPool() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 257
      [SDKD(INFO) 74.83 cbsdk.sdkd.remote remote.py:263] [Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl|Error] Could not init pool. System.NullReferenceException Object reference not set to an instance of an object. at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.CreateSocket() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 279
      [SDKD(INFO) 74.83 cbsdk.sdkd.remote remote.py:263] at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.InitPool() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 257
      [SDKD(INFO) 74.90 cbsdk.sdkd.remote remote.py:263] [Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl|Error] Could not init pool. System.NullReferenceException Object reference not set to an instance of an object. at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.CreateSocket() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 279
      [SDKD(INFO) 74.90 cbsdk.sdkd.remote remote.py:263] at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.InitPool() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 257
      [SDKD(INFO) 74.90 cbsdk.sdkd.remote remote.py:263] [Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl|Error] Could not init pool. System.NullReferenceException Object reference not set to an instance of an object. at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.CreateSocket() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 279
      [SDKD(INFO) 74.90 cbsdk.sdkd.remote remote.py:263] at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.InitPool() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 257
      [SDKD(INFO) 80.13 cbsdk.sdkd.remote remote.py:263] [Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl|Error] Could not init pool. System.NullReferenceException Object reference not set to an instance of an object. at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.CreateSocket() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 279
      [SDKD(INFO) 80.13 cbsdk.sdkd.remote remote.py:263] at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.InitPool() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 257
      [SDKD(INFO) 83.91 cbsdk.sdkd.remote remote.py:263] [Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl|Error] Could not init pool. System.NullReferenceException Object reference not set to an instance of an object. at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.CreateSocket() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 279
      [SDKD(INFO) 83.91 cbsdk.sdkd.remote remote.py:263] at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.InitPool() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 257
      [SDKD(INFO) 83.91 cbsdk.sdkd.remote remote.py:263] [Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl|Error] Could not init pool. System.NullReferenceException Object reference not set to an instance of an object. at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.CreateSocket() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 279
      [SDKD(INFO) 83.91 cbsdk.sdkd.remote remote.py:263] at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.InitPool() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 257
      [SDKD(INFO) 84.24 cbsdk.sdkd.remote remote.py:263] [Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl|Error] Could not init pool. System.NullReferenceException Object reference not set to an instance of an object. at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.CreateSocket() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 279
      [SDKD(INFO) 84.24 cbsdk.sdkd.remote remote.py:263] at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.InitPool() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 257
      [ACTION 85.48 cbsdk.scenario rebalance.py:213] Rebalance action complete
      [ACTION 85.48 cbsdk.scenario base.py:50] Sleeping 5 seconds before collection
      [SDKD(INFO) 88.21 cbsdk.sdkd.remote remote.py:263] [Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl|Error] Could not init pool. System.NullReferenceException Object reference not set to an instance of an object. at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.CreateSocket() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 279
      [SDKD(INFO) 88.21 cbsdk.sdkd.remote remote.py:263] at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.InitPool() in c:\ncbc\lib\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 257
      [DEBUG 90.48 cbsdk.driver driver.py:39] CANCEL:77 @2 {}
      [DEBUG 90.51 cbsdk.driver driver.py:48] CANCEL:77 @2 (OK) {}
      [DEBUG 90.51 cbsdk.driver driver.py:39] CANCEL:78 @3 {}
      [SDKD(INFO) 90.51 cbsdk.sdkd.remote remote.py:263] [Sdkd.Control|Info] Cancelling command for Handle 2
      [DEBUG 90.61 cbsdk.driver driver.py:48] CANCEL:78 @3 (OK) {}
      [DEBUG 90.61 cbsdk.driver driver.py:39] CANCEL:79 @4 {}
      [SDKD(INFO) 90.61 cbsdk.sdkd.remote remote.py:263] [Sdkd.Control|Info] Cancelling command for Handle 3
      [DEBUG 90.63 cbsdk.driver driver.py:48] CANCEL:79 @4 (OK) {}
      [DEBUG 90.63 cbsdk.driver driver.py:39] CANCEL:80 @5 {}
      [SDKD(INFO) 90.64 cbsdk.sdkd.remote remote.py:263] [Sdkd.Control|Info] Cancelling command for Handle 4
      [DEBUG 90.72 cbsdk.driver driver.py:48] CANCEL:80 @5 (OK) {}
      [DEBUG 90.72 cbsdk.driver driver.py:39] CANCEL:81 @6 {}
      [SDKD(INFO) 90.72 cbsdk.sdkd.remote remote.py:263] [Sdkd.Control|Info] Cancelling command for Handle 5
      [DEBUG 90.76 cbsdk.driver driver.py:48] CANCEL:81 @6 (OK) {}
      [DEBUG 90.76 cbsdk.driver driver.py:39] CANCEL:82 @7 {}
      [SDKD(INFO) 90.82 cbsdk.sdkd.remote remote.py:263] [Sdkd.Control|Info] Cancelling command for Handle 6
      [DEBUG 90.88 cbsdk.driver driver.py:48] CANCEL:82 @7 (OK) {}
      [DEBUG 90.88 cbsdk.driver driver.py:39] CANCEL:83 @8 {}
      [SDKD(INFO) 90.89 cbsdk.sdkd.remote remote.py:263] [Sdkd.Control|Info] Cancelling command for Handle 7
      [DEBUG 90.97 cbsdk.driver driver.py:48] CANCEL:83 @8 (OK) {}
      [DEBUG 90.97 cbsdk.driver driver.py:39] CANCEL:84 @9 {}
      [SDKD(INFO) 91.02 cbsdk.sdkd.remote remote.py:263] [Sdkd.Control|Info] Cancelling command for Handle 8
      [DEBUG 91.17 cbsdk.driver driver.py:48] CANCEL:84 @9 (OK) {}
      [DEBUG 91.17 cbsdk.driver driver.py:39] CANCEL:85 @10 {}
      [SDKD(INFO) 91.19 cbsdk.sdkd.remote remote.py:263] [Sdkd.Control|Info] Cancelling command for Handle 9
      [DEBUG 91.22 cbsdk.driver driver.py:48] CANCEL:85 @10 (OK) {}
      [DEBUG 91.22 cbsdk.driver driver.py:39] CANCEL:86 @11 {}
      [SDKD(INFO) 91.24 cbsdk.sdkd.remote remote.py:263] [Sdkd.Control|Info] Cancelling command for Handle 10
      [DEBUG 91.30 cbsdk.driver driver.py:48] CANCEL:86 @11 (OK) {}
      [SDKD(INFO) 91.32 cbsdk.sdkd.remote remote.py:263] [Sdkd.Control|Info] Cancelling command for Handle 11
      [DEBUG 91.78 cbsdk.driver driver.py:48] CB_VIEW_QUERY:14 @2 (OK) {u'Timings': '< Timings Truncated >', u'Summary': {u'0': 507}}
      [DEBUG 92.01 cbsdk.driver driver.py:48] CB_VIEW_QUERY:15 @3 (OK) {u'Timings': '< Timings Truncated >', u'Summary': {u'0': 709}}
      [DEBUG 92.01 cbsdk.driver driver.py:48] CB_VIEW_QUERY:16 @4 (OK) {u'Timings': '< Timings Truncated >', u'Summary': {u'0': 608}}
      [SDKD(INFO) 128.53 cbsdk.sdkd.remote remote.py:263] [Sdkd.ViewQuery|Warn] Unrecognized error System.Net.WebException The operation has timed out at Couchbase.HammockHttpClient.HammockResponseWrapper.ExecuteWith(RestClient client) in c:\ncbc\src\Couchbase\HammockHttpClient.cs:line 141
      [SDKD(INFO) 128.53 cbsdk.sdkd.remote remote.py:263] at Couchbase.HammockHttpClient.HammockRequestWrapper.Couchbase.IHttpRequest.GetResponse() in c:\ncbc\src\Couchbase\HammockHttpClient.cs:line 112
      [SDKD(INFO) 128.54 cbsdk.sdkd.remote remote.py:263] at Couchbase.CouchbaseViewHandler.GetResponse(IDictionary`2 viewParams) in c:\ncbc\src\Couchbase\CouchbaseViewHandler.cs:line 144
      [SDKD(INFO) 128.54 cbsdk.sdkd.remote remote.py:263] at Couchbase.CouchbaseViewHandler.<TransformResults>d__0`1.MoveNext() in c:\ncbc\src\Couchbase\CouchbaseViewHandler.cs:line 38
      [SDKD(INFO) 128.54 cbsdk.sdkd.remote remote.py:263] at Sdkd.Views.ViewCommand.Run() in c:\Users\Administrator\Documents\GitHub\sdkd-net\src\Sdkd\Views\ViewCommand.cs:line 219
      [SDKD(INFO) 139.64 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 139.64 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 139.64 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 139.66 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 140.92 cbsdk.sdkd.remote remote.py:263] [Sdkd.ViewQuery|Warn] Unrecognized error System.Net.WebException The operation has timed out at Couchbase.HammockHttpClient.HammockResponseWrapper.ExecuteWith(RestClient client) in c:\ncbc\src\Couchbase\HammockHttpClient.cs:line 141
      [SDKD(INFO) 140.92 cbsdk.sdkd.remote remote.py:263] at Couchbase.HammockHttpClient.HammockRequestWrapper.Couchbase.IHttpRequest.GetResponse() in c:\ncbc\src\Couchbase\HammockHttpClient.cs:line 112
      [SDKD(INFO) 140.92 cbsdk.sdkd.remote remote.py:263] at Couchbase.CouchbaseViewHandler.GetResponse(IDictionary`2 viewParams) in c:\ncbc\src\Couchbase\CouchbaseViewHandler.cs:line 144
      [SDKD(INFO) 140.92 cbsdk.sdkd.remote remote.py:263] at Couchbase.CouchbaseViewHandler.<TransformResults>d__0`1.MoveNext() in c:\ncbc\src\Couchbase\CouchbaseViewHandler.cs:line 38
      [SDKD(INFO) 140.92 cbsdk.sdkd.remote remote.py:263] at Sdkd.Views.ViewCommand.Run() in c:\Users\Administrator\Documents\GitHub\sdkd-net\src\Sdkd\Views\ViewCommand.cs:line 219
      [SDKD(INFO) 142.35 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 142.36 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 142.38 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 142.41 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 142.46 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 142.49 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 142.55 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 142.57 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 142.62 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 142.63 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 142.66 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 142.68 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 142.71 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 142.75 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 142.77 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 142.79 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 142.82 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 142.87 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 142.91 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 142.94 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 142.99 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 143.02 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 143.05 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 143.10 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 143.13 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 143.19 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 143.25 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 143.30 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 143.35 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 143.36 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 143.39 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 143.43 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 143.47 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 143.51 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 143.53 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 143.63 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 143.68 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 143.69 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 143.71 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 143.73 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 143.77 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 143.80 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 143.85 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 143.90 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 143.94 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 143.96 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 143.99 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 144.04 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 144.04 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 144.05 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 144.10 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 144.14 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 144.17 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 144.20 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 144.23 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 144.28 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 144.30 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 144.32 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 144.34 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 144.37 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 144.39 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 144.43 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 144.49 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 144.52 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 144.54 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 144.57 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 144.62 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 144.63 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 144.65 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 144.66 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 144.72 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 144.76 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 144.80 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 144.82 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 144.84 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 144.89 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 144.91 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 144.93 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 144.98 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 144.98 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 145.01 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 145.03 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 145.07 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 145.09 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 145.11 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 145.13 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 145.18 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 145.18 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 145.21 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 145.22 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 145.24 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 145.29 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 145.30 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 145.31 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 145.35 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 145.38 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 145.40 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 145.43 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 145.49 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 145.51 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 145.55 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 145.58 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 145.62 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 145.64 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 145.66 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 145.68 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 145.72 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 145.76 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 145.81 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 145.84 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 145.90 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 145.94 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 145.96 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 145.97 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 146.01 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 146.06 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 146.09 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 146.12 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 146.15 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 146.18 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 146.21 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 146.24 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 146.27 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 146.27 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 146.32 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 146.36 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 146.39 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 146.44 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 146.46 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 146.50 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 146.52 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 146.54 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 146.58 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 146.63 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 146.68 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 146.70 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 146.79 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 146.82 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 146.84 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 146.85 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 146.88 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 146.91 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 146.94 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 146.97 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 147.04 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 147.08 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 147.13 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 147.16 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 147.21 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 147.22 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 147.24 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 147.25 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 147.32 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 147.37 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 147.40 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 147.44 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 147.51 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 147.55 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 147.57 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 147.60 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 147.61 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 147.68 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 147.71 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 147.73 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 147.80 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 147.91 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 147.98 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 148.02 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 148.05 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 148.07 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 148.08 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 148.12 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 148.18 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 148.22 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 148.24 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 148.27 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 148.30 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 148.32 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 148.33 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 148.33 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 148.41 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 148.43 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 148.44 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 148.46 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 148.54 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 148.56 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 148.57 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 148.62 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 148.63 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 148.68 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 148.72 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 148.76 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 148.79 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 148.83 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 148.84 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 148.86 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 148.87 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 148.91 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 148.96 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 148.98 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 149.03 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 149.09 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 149.13 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 149.13 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 149.17 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 149.22 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 149.23 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 149.25 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 149.30 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 149.34 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 149.36 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 149.42 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 149.45 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 149.50 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 149.53 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 149.55 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 149.56 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 149.61 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 149.63 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 149.64 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 149.68 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 149.72 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 149.75 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 149.76 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 149.79 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 149.83 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 149.84 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 149.89 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 149.94 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 149.99 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 150.03 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 150.06 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 150.11 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 150.14 cbsdk.sdkd.remote remote.py:263] [Couchbase.HammockHttpClient|Warn] Retrying
      [SDKD(INFO) 150.17 cbsdk.sdkd.remote remote.py:263] [Sdkd.Main|Fatal] Requested suppression of exception dialog box..
      [SDKD(INFO) 150.22 cbsdk.sdkd.remote remote.py:263] [Sdkd.Main|Fatal] Unhandled Exception System.NullReferenceException Object reference not set to an instance of an object. at Hammock.RestClient.CompleteWithQuery(WebQuery query, RestRequest request, RestCallback callback, WebQueryAsyncResult result) in C:\Users\Daniel\Desktop\Dropbox_7_Source_Code_1_Projects\hammock\src\net35\Hammock\RestClient.cs:line 1797
      [SDKD(INFO) 150.22 cbsdk.sdkd.remote remote.py:263] at Hammock.RestClient.<>c_DisplayClass18.<BeginRequestImpl>b_15(Object sender, WebQueryResponseEventArgs args) in C:\Users\Daniel\Desktop\Dropbox_7_Source_Code_1_Projects\hammock\src\net35\Hammock\RestClient.cs:line 1190
      [SDKD(INFO) 150.22 cbsdk.sdkd.remote remote.py:263] at System.EventHandler`1.Invoke(Object sender, TEventArgs e)
      [SDKD(INFO) 150.22 cbsdk.sdkd.remote remote.py:263] at Hammock.Web.WebQuery.OnQueryResponse(WebQueryResponseEventArgs args) in C:\Users\Daniel\Desktop\Dropbox_7_Source_Code_1_Projects\hammock\src\net35\Hammock\Web\WebQuery.cs:line 991
      [SDKD(INFO) 150.22 cbsdk.sdkd.remote remote.py:263] at Hammock.Web.WebQuery.HandleWebException(WebException exception) in C:\Users\Daniel\Desktop\Dropbox_7_Source_Code_1_Projects\hammock\src\net35\Hammock\Web\WebQuery.cs:line 831
      [SDKD(INFO) 150.22 cbsdk.sdkd.remote remote.py:263] at Hammock.Web.WebQuery.GetAsyncResponseCallback(IAsyncResult asyncResult) in C:\Users\Daniel\Desktop\Dropbox_7_Source_Code_1_Projects\hammock\src\net35\Hammock\Web\WebQuery.Async.cs:line 396
      [SDKD(INFO) 150.22 cbsdk.sdkd.remote remote.py:263] at System.Net.LazyAsyncResult.Complete(IntPtr userToken)
      [SDKD(INFO) 150.23 cbsdk.sdkd.remote remote.py:263] at System.Net.ContextAwareResult.CompleteCallback(Object state)
      [SDKD(INFO) 150.23 cbsdk.sdkd.remote remote.py:263] at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean ignoreSyncCtx)
      [SDKD(INFO) 150.23 cbsdk.sdkd.remote remote.py:263] at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
      [SDKD(INFO) 150.23 cbsdk.sdkd.remote remote.py:263] at System.Net.ContextAwareResult.Complete(IntPtr userToken)
      [SDKD(INFO) 150.23 cbsdk.sdkd.remote remote.py:263] at System.Net.LazyAsyncResult.ProtectedInvokeCallback(Object result, IntPtr userToken)
      [SDKD(INFO) 150.23 cbsdk.sdkd.remote remote.py:263] at System.Net.HttpWebRequest.SetResponse(Exception E)
      [SDKD(INFO) 150.23 cbsdk.sdkd.remote remote.py:263] at System.Net.HttpWebRequest.Abort(Exception exception, Int32 abortState)
      [SDKD(INFO) 150.23 cbsdk.sdkd.remote remote.py:263] at System.Net.HttpWebRequest.Abort()
      [SDKD(INFO) 150.23 cbsdk.sdkd.remote remote.py:263] at Hammock.Web.WebQuery.TimedOutCallback(Object state, Boolean timedOut) in C:\Users\Daniel\Desktop\Dropbox_7_Source_Code_1_Projects\hammock\src\net35\Hammock\Web\WebQuery.Async.cs:line 267
      [SDKD(INFO) 150.23 cbsdk.sdkd.remote remote.py:263] at System.Threading._ThreadPoolWaitOrTimerCallback.WaitOrTimerCallback_Context(Object state, Boolean timedOut)
      [SDKD(INFO) 150.23 cbsdk.sdkd.remote remote.py:263] at System.Threading._ThreadPoolWaitOrTimerCallback.WaitOrTimerCallback_Context_t(Object state)
      [SDKD(INFO) 150.23 cbsdk.sdkd.remote remote.py:263] at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean ignoreSyncCtx)
      [SDKD(INFO) 150.23 cbsdk.sdkd.remote remote.py:263] at System.Threading._ThreadPoolWaitOrTimerCallback.PerformWaitOrTimerCallback(Object state, Boolean timedOut)
      [SDKD(INFO) 150.23 cbsdk.sdkd.remote remote.py:263] [Sdkd.Main|Fatal] Will now exit..

      1. NCBC-172-readable.txt
        3 kB
        Matt Ingenthron
      2. RestSharp.dll
        147 kB
        John Zablocki
      3. RestSharpHttpClient.cs
        4 kB
        John Zablocki
      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        mnunberg Mark Nunberg added a comment -

        I also get this when a node is being shut down

        Show
        mnunberg Mark Nunberg added a comment - I also get this when a node is being shut down
        Hide
        ingenthr Matt Ingenthron added a comment -

        Setting this to 1.2 for now, but let's chat about this in the morning John. Thanks.

        Show
        ingenthr Matt Ingenthron added a comment - Setting this to 1.2 for now, but let's chat about this in the morning John. Thanks.
        Hide
        john John Zablocki (Inactive) added a comment -

        Mark, you can use the attached client file and dll to test whether a different provider handles better the failover scenario.

        Show
        john John Zablocki (Inactive) added a comment - Mark, you can use the attached client file and dll to test whether a different provider handles better the failover scenario.
        Hide
        john John Zablocki (Inactive) added a comment -

        And for config:

        var config = new CouchbaseClientConfiguration();
        config.Bucket = "beer-sample";
        config.BucketPassword = "";
        config.Urls.Add(new Uri("http://localhost:8091/pools"));
        config.HttpClientFactory = new RestSharpHttpClientFactory();

        //Quick test of Store/Get operations
        var client = new CouchbaseClient(config);

        Show
        john John Zablocki (Inactive) added a comment - And for config: var config = new CouchbaseClientConfiguration(); config.Bucket = "beer-sample"; config.BucketPassword = ""; config.Urls.Add(new Uri("http://localhost:8091/pools")); config.HttpClientFactory = new RestSharpHttpClientFactory(); //Quick test of Store/Get operations var client = new CouchbaseClient(config);
        Hide
        mnunberg Mark Nunberg added a comment -

        Running them now. It took me about an hour to figure out how to build it; a git branch or commit would be helpful in the future

        Show
        mnunberg Mark Nunberg added a comment - Running them now. It took me about an hour to figure out how to build it; a git branch or commit would be helpful in the future
        Hide
        mnunberg Mark Nunberg added a comment -

        Reports are looking good so far. However I'm still seeing inconclusive results for some tests where they just 'hang' – is there a specific timeout that we have somewhere –

        I'll run the tests a bit longer now (specifically, at least 90 seconds after rebalance/failover/whatever) - I should expect everything to recover to normal levels by then; right?

        Show
        mnunberg Mark Nunberg added a comment - Reports are looking good so far. However I'm still seeing inconclusive results for some tests where they just 'hang' – is there a specific timeout that we have somewhere – I'll run the tests a bit longer now (specifically, at least 90 seconds after rebalance/failover/whatever) - I should expect everything to recover to normal levels by then; right?
        Hide
        mnunberg Mark Nunberg added a comment -

        Passes all tests. Nice work!

        • This version is much better
        Show
        mnunberg Mark Nunberg added a comment - Passes all tests. Nice work! This version is much better
        Hide
        john John Zablocki (Inactive) added a comment -

        I will fix up the code and add this as the default, leaving Hammock as an option so that it doesn't break anyone.

        Show
        john John Zablocki (Inactive) added a comment - I will fix up the code and add this as the default, leaving Hammock as an option so that it doesn't break anyone.
        Hide
        john John Zablocki (Inactive) added a comment -

        Attached to this case is a signed, pre-release build of 1.2.1.

        Show
        john John Zablocki (Inactive) added a comment - Attached to this case is a signed, pre-release build of 1.2.1.
        Hide
        skumar Saran Kumar (Inactive) added a comment -

        Hi John/Mark,

        As mentioned by Hari in CBSE-304, i have provided Couchbase-Net-Client-1.2.1.zip to the customer(Hobsons). After testing, he is stating that the issue is not fixed and the code fails the same exact way as before. Could you please look into this.

        Thanks,
        Saran

        Show
        skumar Saran Kumar (Inactive) added a comment - Hi John/Mark, As mentioned by Hari in CBSE-304, i have provided Couchbase-Net-Client-1.2.1.zip to the customer(Hobsons). After testing, he is stating that the issue is not fixed and the code fails the same exact way as before. Could you please look into this. Thanks, Saran
        Hide
        mnunberg Mark Nunberg added a comment -

        This issue does not seem to have anything to do with what is listed in CBSE-304. This issue is specifically about cluster changes during view operations; the issue mentioned in CBSE-304 seems to deal with providing CAS inside an arithmetic operation..

        Show
        mnunberg Mark Nunberg added a comment - This issue does not seem to have anything to do with what is listed in CBSE-304. This issue is specifically about cluster changes during view operations; the issue mentioned in CBSE-304 seems to deal with providing CAS inside an arithmetic operation..
        Hide
        john John Zablocki (Inactive) added a comment -

        This is an unrelated fix.

        Show
        john John Zablocki (Inactive) added a comment - This is an unrelated fix.
        Hide
        john John Zablocki (Inactive) added a comment -

        Released

        Show
        john John Zablocki (Inactive) added a comment - Released

          People

          • Assignee:
            john John Zablocki (Inactive)
            Reporter:
            mnunberg Mark Nunberg
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes