Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-39535

simple-test on cheshire-cat silently fails

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: Cheshire-Cat
    • Fix Version/s: 7.0.0
    • Component/s: test-execution
    • Labels:
      None
    • Triage:
      Untriaged
    • Story Points:
      1
    • Is this a Regression?:
      Unknown

      Description

      Essentially the problem here is that after a Stack trace(indicating some error) but the exit code is 0. It is expected that make simple-test fails if we fail a test. 

       

      b'[2020-05-21 23:41:34,628] - [rest_client:943] ERROR - socket error while connecting to http://127.0.0.1:9001/pools/default/buckets/default?basic_stats=true error [Errno 24] Too many open files '
      b'[2020-05-21 23:41:37,631] - [rest_client:943] ERROR - socket error while connecting to http://127.0.0.1:9001/pools/default/buckets/default?basic_stats=true error [Errno 24] Too many open files '
      b'[2020-05-21 23:41:40,632] - [rest_client:943] ERROR - socket error while connecting to http://127.0.0.1:9001/pools/default/buckets/default?basic_stats=true error [Errno 24] Too many open files '
      b'[2020-05-21 23:43:34,758] - [rest_client:945] ERROR - Tried ta connect 41 times'
      b'[2020-05-21 23:43:34,759] - [data_helper:863] WARNING - unable to establish connection to 127.0.0.1. cleanup open connections'
      b'[2020-05-21 23:43:34,759] - [task:375] ERROR - Unexpected error: unable to reach the host @ 127.0.0.1'
      b'[2020-05-21 23:43:34,760] - [task:376] WARNING - vbucket map not ready after try 5'
      ERROR======================================================================
      ERROR: suite_tearDown (xdcr.uniXDCR.unidirectional)
      ----------------------------------------------------------------------
      Traceback (most recent call last):
        File "lib/membase/api/rest_client.py", line 913, in _http_request
          params, headers)
        File "lib/membase/api/httplib2/__init__.py", line 1953, in request
        File "lib/membase/api/httplib2/__init__.py", line 1618, in _request
        File "lib/membase/api/httplib2/__init__.py", line 1524, in _conn_request
        File "lib/membase/api/httplib2/__init__.py", line 1220, in connect
        File "lib/membase/api/httplib2/__init__.py", line 1170, in connect
        File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/socket.py", line 151, in __init__
      OSError: [Errno 24] Too many open filesDuring handling of the above exception, another exception occurred:Traceback (most recent call last):
        File "pytests/xdcr/uniXDCR.py", line 18, in setUp
          super(unidirectional, self).setUp()
        File "pytests/xdcr/xdcrnewbasetests.py", line 2653, in setUp
          self.__setup_for_test()
        File "pytests/xdcr/xdcrnewbasetests.py", line 2768, in __setup_for_test
          self.__create_buckets()
        File "pytests/xdcr/xdcrnewbasetests.py", line 3015, in __create_buckets
          maxttl=maxttl)
        File "pytests/xdcr/xdcrnewbasetests.py", line 1435, in create_default_bucket
          self.__clusterop.create_default_bucket(bucket_params)
        File "lib/couchbase_helper/cluster.py", line 308, in create_default_bucket
        File "lib/tasks/future.py", line 160, in result
        File "lib/tasks/future.py", line 112, in __get_result
        File "lib/tasks/task.py", line 363, in check
          if BucketOperationHelper.wait_for_memcached(self.server, self.bucket, self.alt_addr):
        File "lib/membase/helper/bucket_helper.py", line 344, in wait_for_memcached
        File "lib/membase/helper/bucket_helper.py", line 268, in wait_for_vbuckets_ready_state
        File "lib/memcached/helper/data_helper.py", line 774, in __init__
        File "lib/memcached/helper/data_helper.py", line 780, in reset
        File "lib/memcached/helper/data_helper.py", line 836, in request_map
        File "lib/memcached/helper/data_helper.py", line 865, in add_memcached
        File "lib/memcached/helper/data_helper.py", line 858, in add_memcached
        File "lib/memcached/helper/data_helper.py", line 315, in direct_client
        File "lib/membase/api/rest_client.py", line 2444, in get_bucket
          status, content, header = self._http_request(api)
        File "lib/membase/api/rest_client.py", line 946, in _http_request
          raise ServerUnavailableException(ip=self.ip)
      membase.api.exception.ServerUnavailableException: unable to reach the host @ 127.0.0.1----------------------------------------------------------------------
      Ran 1 test in 156.683sFAILED (errors=1)
      b'[<FrameSummary file /usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py, line 890 in _bootstrap>, <FrameSummary file /usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py, line 926 in _bootstrap_inner>, <FrameSummary file lib/tasks/taskmanager.py, line 31 in run>, <FrameSummary file lib/tasks/task.py, line 95 in step>, <FrameSummary file lib/tasks/task.py, line 379 in check>, <FrameSummary file lib/tasks/future.py, line 265 in set_exception>]'
      b'Thu May 21 23:43:34 2020'
      b'[<FrameSummary file testrunner.py, line 1029 in <module>>, <FrameSummary file testrunner.py, line 1012 in watcher>, <FrameSummary file testrunner.py, line 984 in main>, <FrameSummary file testrunner.py, line 756 in runtests>, <FrameSummary file /usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/unittest/runner.py, line 176 in run>, <FrameSummary file /usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/unittest/suite.py, line 84 in __call__>, <FrameSummary file /usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/unittest/suite.py, line 122 in run>, <FrameSummary file /usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/unittest/case.py, line 676 in __call__>, <FrameSummary file /usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/unittest/case.py, line 624 in run>, <FrameSummary file pytests/xdcr/uniXDCR.py, line 18 in setUp>, <FrameSummary file pytests/xdcr/xdcrnewbasetests.py, line 2653 in setUp>, <FrameSummary file pytests/xdcr/xdcrnewbasetests.py, line 2768 in __setup_for_test>, <FrameSummary file pytests/xdcr/xdcrnewbasetests.py, line 3015 in __create_buckets>, <FrameSummary file pytests/xdcr/xdcrnewbasetests.py, line 1435 in create_default_bucket>, <FrameSummary file lib/couchbase_helper/cluster.py, line 308 in create_default_bucket>, <FrameSummary file lib/tasks/future.py, line 160 in result>, <FrameSummary file lib/tasks/future.py, line 111 in __get_result>]'
      b"('rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops', ' pass')"
      b"('rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops', ' pass')"
      b"('memcapable.WarmUpMemcachedTest.do_warmup_100k', ' pass')"
      b"('view.createdeleteview.CreateDeleteViewTests.test_view_ops', ' pass')"
      b"('view.viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_in', ' pass')"
      b"('view.viewquerytests.ViewQueryTests.test_simple_dataset_stale_queries_data_modification', ' pass')"
      b"('xdcr.uniXDCR.unidirectional.load_with_ops', ' pass')"
      b"('xdcr.uniXDCR.unidirectional.load_with_failover', ' pass')"
      b'During the test, Remote Connections: 0, Disconnections: 236'
      b"('Thread', <TaskManager(Cluster_Thread, started 123145513283584)>, 'was not properly terminated, will be terminated now.')"
      b'Shutting down the thread...'
      b"('Thread', <TaskManager(Cluster_Thread, started 123145518538752)>, 'was not properly terminated, will be terminated now.')"
      b'Shutting down the thread...'
      b"('Thread', <TaskManager(Cluster_Thread, started 123145523793920)>, 'was not properly terminated, will be terminated now.')"
      b'Shutting down the thread...'
      b'*** TestRunner ***'
      INFO:root:stopping node 0
      INFO:root:stopping node 1
      INFO:root:stopping node 2
      INFO:root:stopping node 3

       
      Exits with zero,

      $ echo $?
      0  

        Attachments

        For Gerrit Dashboard: MB-39535
        # Subject Branch Project Status CR V

          Activity

          Hide
          drigby Dave Rigby added a comment -

          After patch I'm seeing kv_engine-master-post-commit fail on every build starting at: http://cv.jenkins.couchbase.com/job/kv_engine-master-post-commit/3886/

          Error message is somewhat cryptic - all the tests report "pass" but the overall testrunner fails:

          00:23:59.332 b"('rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops', ' pass')"
          00:23:59.332 b"('rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops', ' pass')"
          00:23:59.332 b"('memcapable.WarmUpMemcachedTest.do_warmup_100k', ' pass')"
          00:23:59.332 b"('view.createdeleteview.CreateDeleteViewTests.test_view_ops', ' pass')"
          00:23:59.332 b"('view.viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_in', ' pass')"
          00:23:59.332 b"('view.viewquerytests.ViewQueryTests.test_simple_dataset_stale_queries_data_modification', ' pass')"
          00:23:59.332 b"('xdcr.uniXDCR.unidirectional.load_with_ops', ' pass')"
          00:23:59.332 b"('xdcr.uniXDCR.unidirectional.load_with_failover', ' pass')"
          00:23:59.332 b'During the test, Remote Connections: 0, Disconnections: 236'
          00:23:59.332 b"('Thread', <TaskManager(Cluster_Thread, started 140162563655424)>, 'was not properly terminated, will be terminated now.')"
          00:23:59.332 b'Shutting down the thread...'
          00:23:59.332 b"('Thread', <TaskManager(Cluster_Thread, started 140162521458432)>, 'was not properly terminated, will be terminated now.')"
          00:23:59.332 b'Shutting down the thread...'
          00:23:59.332 b'*** TestRunner ***'
          00:23:59.353 Makefile:28: recipe for target 'simple-test' failed
          00:23:59.353 make: *** [simple-test] Error 1
          

          Show
          drigby Dave Rigby added a comment - After patch I'm seeing kv_engine-master-post-commit fail on every build starting at: http://cv.jenkins.couchbase.com/job/kv_engine-master-post-commit/3886/ Error message is somewhat cryptic - all the tests report "pass" but the overall testrunner fails: 00:23:59.332 b"('rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops', ' pass')" 00:23:59.332 b"('rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops', ' pass')" 00:23:59.332 b"('memcapable.WarmUpMemcachedTest.do_warmup_100k', ' pass')" 00:23:59.332 b"('view.createdeleteview.CreateDeleteViewTests.test_view_ops', ' pass')" 00:23:59.332 b"('view.viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_in', ' pass')" 00:23:59.332 b"('view.viewquerytests.ViewQueryTests.test_simple_dataset_stale_queries_data_modification', ' pass')" 00:23:59.332 b"('xdcr.uniXDCR.unidirectional.load_with_ops', ' pass')" 00:23:59.332 b"('xdcr.uniXDCR.unidirectional.load_with_failover', ' pass')" 00:23:59.332 b'During the test, Remote Connections: 0, Disconnections: 236' 00:23:59.332 b"('Thread', <TaskManager(Cluster_Thread, started 140162563655424)>, 'was not properly terminated, will be terminated now.')" 00:23:59.332 b'Shutting down the thread...' 00:23:59.332 b"('Thread', <TaskManager(Cluster_Thread, started 140162521458432)>, 'was not properly terminated, will be terminated now.')" 00:23:59.332 b'Shutting down the thread...' 00:23:59.332 b'*** TestRunner ***' 00:23:59.353 Makefile:28: recipe for target 'simple-test' failed 00:23:59.353 make: *** [simple-test] Error 1
          Hide
          drigby Dave Rigby added a comment -

          Bumping this to critical as the initial fix for it has made all kv_engine-master-post-commit jobs start failing.

          Show
          drigby Dave Rigby added a comment - Bumping this to critical as the initial fix for it has made all kv_engine-master-post-commit jobs start failing.
          Hide
          raju Raju Suravarjjala added a comment -

          Ritesh Agarwal Can you please take a look at it?

          Show
          raju Raju Suravarjjala added a comment - Ritesh Agarwal Can you please take a look at it?
          Hide
          steve.watanabe Steve Watanabe added a comment -

          Here's a simple-test failure on a gerrit review with effectively no change: http://cv.jenkins.couchbase.com/job/ns-server-simple-test/1463/console

          Show
          steve.watanabe Steve Watanabe added a comment - Here's a simple-test failure on a gerrit review with effectively no change: http://cv.jenkins.couchbase.com/job/ns-server-simple-test/1463/console
          Hide
          Abhijeeth.Nuthan Abhijeeth Nuthan added a comment -

          Edited the bug description as the ulimit problem was a user error on my part. 

          Show
          Abhijeeth.Nuthan Abhijeeth Nuthan added a comment - Edited the bug description as the ulimit problem was a user error on my part. 

            People

            Assignee:
            ritesh.agarwal Ritesh Agarwal
            Reporter:
            Abhijeeth.Nuthan Abhijeeth Nuthan
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Gerrit Reviews

                There are no open Gerrit changes

                  PagerDuty