Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-47050

[System Test] PANIC observed in System Test

    XMLWordPrintable

Details

    • Untriaged
    • Centos 64-bit
    • 1
    • No

    Description

      Build : 7.0.0-5295
      Test : -test tests/2i/cheshirecat/test_idx_clusterops_cheshire_cat_recovery.yml -scope tests/2i/cheshirecat/scope_idx_cheshire_cat_dgm.yml -scale 2
      Scale : 3
      Iteration : 9th
      Day : 4th

      Panic observed on Indexer at *.216

      No Indexer crash observed.

      2021-06-21T19:25:14.235-07:00 [Info] MutationStreamReader:Shutdown StreamReader INIT_STREAM
      		 
      2021-06-21T19:25:14.237-07:00 [Error] DATP[->dataport ":9103"] Accept() Error: accept tcp4 0.0.0.0:9103: use of closed network connection
      		 
      2021-06-21T19:25:14.237-07:00 [Info] DATP[->dataport ":9103"] ... stopped
      		 
      2021-06-21T19:25:14.237-07:00 [Error] DATP[->dataport ":9103"] gen-server crashed: assignment to entry in nil map
      		 
      2021-06-21T19:25:14.237-07:00 [Error]   /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:312 +0x22
      		 
      github.com/couchbase/indexing/secondary/dataport.(*Server).genServer.func1(0xc015190600)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dataport/server.go:245 +0xf7
      		 
      panic(0x10e9b60, 0x14b8f90)
      		 
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/panic.go:679 +0x1b2
      		 
      github.com/couchbase/indexing/secondary/dataport.(*Server).genServer.func3(0xc01c08ed00, 0x1, 0x1)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dataport/server.go:329 +0x36d
      		 
      github.com/couchbase/indexing/secondary/dataport.(*Server).genServer(0xc015190600, 0xc022dca960, 0xc022dca9c0)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dataport/server.go:372 +0x95b
      		 
      created by github.com/couchbase/indexing/secondary/dataport.NewServer
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dataport/server.go:185 +0x628
      		 
       
      		 
       
      		 
      2021-06-21T19:25:14.237-07:00 [Error] DATP[->dataport ":9103"] handleClose() crashed: close of closed channel
      		 
      2021-06-21T19:25:14.238-07:00 [Error]   /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:312 +0x22
      		 
      github.com/couchbase/indexing/secondary/dataport.(*Server).handleClose.func1(0xc015190600)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dataport/server.go:409 +0xf7
      		 
      panic(0x10e9b60, 0x14b90d0)
      		 
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/panic.go:679 +0x1b2
      		 
      github.com/couchbase/indexing/secondary/dataport.(*Server).handleClose(0xc015190600)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dataport/server.go:424 +0x1bb
      		 
      github.com/couchbase/indexing/secondary/dataport.(*Server).genServer.func1(0xc015190600)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dataport/server.go:246 +0x173
      		 
      panic(0x10e9b60, 0x14b8f90)
      		 
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/panic.go:679 +0x1b2
      		 
      github.com/couchbase/indexing/secondary/dataport.(*Server).genServer.func3(0xc01c08ed00, 0x1, 0x1)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dataport/server.go:329 +0x36d
      		 
      github.com/couchbase/indexing/secondary/dataport.(*Server).genServer(0xc015190600, 0xc022dca960, 0xc022dca9c0)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dataport/server.go:372 +0x95b
      		 
      created by github.com/couchbase/indexing/secondary/dataport.NewServer
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dataport/server.go:185 +0x628
      		 
       
      		 
       
      		 
      2021-06-21T19:25:14.239-07:00 [Info] Timekeeper::handleRemoveKeyspaceFromStream 
      		 
              Message: MsgStreamUpdate
      		 
              Type: REMOVE_KEYSPACE_FROM_STREAM
      		 
              Stream: INIT_STREAM
      		 
              KeyspaceId: bucket4:_default:U05XEHaY7h8cBao6RJcK03V
      		 
              BuildTS: []
      		 
              IndexList: []
      		 
              Async: false
      		 
              SessionId: 1
      		 
              CollectionId: 
      		 
              CollectionAware: false
      		 
              EnableOSO: false
      		 
              RestartTs: <nil>
      		 
      2021-06-21T19:25:14.239-07:00 [Info] Timekeeper::removeKeyspaceFromStream remove index 1945269381096944475 from stream INIT_STREAM keyspaceId bucket4:_default:U05XEHaY7h8cBao6RJcK03V
      		 
      2021-06-21T19:25:14.239-07:00 [Info] Timekeeper::stopTimer INIT_STREAM bucket4:_default:U05XEHaY7h8cBao6RJcK03V
      		 
      2021-06-21T19:25:14.239-07:00 [Info] StreamState::cleanupKeyspaceIdFromStream KeyspaceId bucket4:_default:U05XEHaY7h8cBao6RJcK03V Deleted from Stream INIT_STREAM
      		 
      2021-06-21T19:25:14.239-07:00 [Info] MutationStreamReader::handleUpdateKeyspaceStatsMap
      		 
      2021-06-21T19:25:14.239-07:00 [Info] MutationMgr::handleUpdateKeyspaceStatsMap Stream MAINT_STREAM Succceed

       

      Index nodes - 172.23.107.2, 172.23.107.3, 172.23.107.4, 172.23.107.5, 172.23.97.216

      Logs are available here - 

      cbcollect logs:

               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1624333623/collectinfo-2021-06-22T034704-ns_1%40172.23.107.2.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1624333623/collectinfo-2021-06-22T034704-ns_1%40172.23.107.3.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1624333623/collectinfo-2021-06-22T034704-ns_1%40172.23.107.4.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1624333623/collectinfo-2021-06-22T034704-ns_1%40172.23.107.5.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1624333623/collectinfo-2021-06-22T034704-ns_1%40172.23.97.215.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1624333623/collectinfo-2021-06-22T034704-ns_1%40172.23.97.216.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1624333623/collectinfo-2021-06-22T034704-ns_1%40172.23.97.227.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1624333623/collectinfo-2021-06-22T034704-ns_1%40172.23.97.232.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1624333623/collectinfo-2021-06-22T034704-ns_1%40172.23.97.235.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1624333623/collectinfo-2021-06-22T034704-ns_1%40172.23.97.236.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1624333623/collectinfo-2021-06-22T034704-ns_1%40172.23.97.237.zip

       

      Attachments

        Issue Links

          backports to

          Bug - A problem which impairs or prevents the functions of the product. MB-47634 [BP 7.0.2 MB-47050] - [System Test] PANIC observed in System Test

          • Major - Major loss of function.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              varun.velamuri Varun Velamuri
              hemant.rajput Hemant Rajput
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty