Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-43416

[BP 6.0.5] - goxdcr exit with status 2 and panic

    XMLWordPrintable

Details

    • Untriaged
    • No

    Description

      Build 6.5.0-2082

      In high bucket density test(with 30 buckets) with replication on, we observed that goxdcr process on source cluster kv node exit multiple times with status 2 and below panic.
      This happened when we hard failed over one kv node and started rebalance without adding that node back in cluster.
      Destination cluster does not show any errors.

      Logs of KV nodes in source cluster:
      https://s3.amazonaws.com/bugdb/jira/mh_high_bkt_density_failover/collectinfo-2019-01-23T055648-ns_1%40172.23.97.12.zip
      https://s3.amazonaws.com/bugdb/jira/mh_high_bkt_density_failover/collectinfo-2019-01-23T055648-ns_1%40172.23.97.13.zip

      Panic:

      Service 'goxdcr' exited with status 2. Restarting. Messages:
       2019-01-22T22:39:17.343-08:00 INFO GOXDCR.DcpNozzle: dcp_a5944500fffd312188a4487bf14b5802/bucket-7/bucket-7_172.23.97.13:11210_0: startUprStreams for [87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427]...
       panic: runtime error: invalid memory address or nil pointer dereference
       [signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0x83aa0e]

      Attachments

        Issue Links

          For Gerrit Dashboard: MB-43416
          # Subject Branch Project Status CR V

          Activity

            neil.huang Neil Huang created issue -
            neil.huang Neil Huang made changes -
            Field Original Value New Value
            Link This issue Clones MB-32780 [ MB-32780 ]
            neil.huang Neil Huang made changes -
            Assignee Mahesh Mandhare [ mahesh.mandhare ] Neil Huang [ neil.huang ]
            neil.huang Neil Huang made changes -
            Affects Version/s 6.0.5 [ 16729 ]
            Affects Version/s 6.5.0 [ 15037 ]
            neil.huang Neil Huang made changes -
            Fix Version/s 6.0.5 [ 16729 ]
            Fix Version/s 6.5.0 [ 15037 ]
            neil.huang Neil Huang made changes -
            Priority Major [ 3 ] Critical [ 2 ]
            neil.huang Neil Huang made changes -
            Priority Critical [ 2 ] Major [ 3 ]
            neil.huang Neil Huang added a comment -

            Not sure if backport is the right option yet. Closing this bug in the interim.

            neil.huang Neil Huang added a comment - Not sure if backport is the right option yet. Closing this bug in the interim.
            neil.huang Neil Huang made changes -
            Resolution Incomplete [ 4 ]
            Status Open [ 1 ] Closed [ 6 ]
            neil.huang Neil Huang made changes -
            Resolution Incomplete [ 4 ]
            Status Closed [ 6 ] Reopened [ 4 ]
            neil.huang Neil Huang made changes -
            Link This issue is a backport of MB-32780 [ MB-32780 ]
            neil.huang Neil Huang made changes -
            Link This issue Clones MB-32780 [ MB-32780 ]
            neil.huang Neil Huang added a comment -

            Reopening the bug for backporting

            neil.huang Neil Huang added a comment - Reopening the bug for backporting
            neil.huang Neil Huang made changes -
            Link This issue is duplicated by MB-43413 [ MB-43413 ]
            neil.huang Neil Huang added a comment -

            Raising to critical to reflect that of original bug

            neil.huang Neil Huang added a comment - Raising to critical to reflect that of original bug
            neil.huang Neil Huang made changes -
            Priority Major [ 3 ] Critical [ 2 ]
            jliang John Liang made changes -
            Link This issue blocks MB-40511 [ MB-40511 ]
            neil.huang Neil Huang made changes -
            Assignee Neil Huang [ neil.huang ] Arunkumar Senthilnathan [ arunkumar ]
            Resolution Fixed [ 1 ]
            Status Reopened [ 4 ] Resolved [ 5 ]

            Build couchbase-server-6.0.5-3337 contains goxdcr commit a85b3c4 with commit message:
            MB-43416 - do not delete vbHanshakeHelper as part of DCP nozzle stop to avoid panic in case start() lags

            build-team Couchbase Build Team added a comment - Build couchbase-server-6.0.5-3337 contains goxdcr commit a85b3c4 with commit message: MB-43416 - do not delete vbHanshakeHelper as part of DCP nozzle stop to avoid panic in case start() lags
            wayne Wayne Siu made changes -
            Labels high-bucket-density approved-for-6.0.5 high-bucket-density

            Build couchbase-server-6.0.5-3338 contains goxdcr commit d331735 with commit message:
            MB-43416 - fix cherry-pick compilation error

            build-team Couchbase Build Team added a comment - Build couchbase-server-6.0.5-3338 contains goxdcr commit d331735 with commit message: MB-43416 - fix cherry-pick compilation error

            Issue not seen in 6.0.5-3338

            arunkumar Arunkumar Senthilnathan added a comment - Issue not seen in 6.0.5-3338
            arunkumar Arunkumar Senthilnathan made changes -
            Status Resolved [ 5 ] Closed [ 6 ]

            People

              arunkumar Arunkumar Senthilnathan
              neil.huang Neil Huang
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty