Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-41471

Upgrading to MB-41255

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Not a Bug
    • 6.0.4, 6.5.1, 6.6.1
    • None
    • couchbase-bucket
    • None
    • 6.0.4-3082 --> 6.0.4-3098(MP2)

    Description

      Updated MB as this issue is generic to any upgrade where we want to get a user to a release with MB-41255, this could be 6.0.4-MP2, 6.5.1 (an MP planned) or 6.6.1

      Steps to Repro
      Ran the following script on 6.0.4-3082.

      ./testrunner -i /tmp/win10-bucket-ops.ini -p  -t epengine.basic_ops.basic_ops.test_MB_32114,nodes_init=3,replicas=2,skip_cleanup=True,bucket_size=100,items=200000,active_resident_threshold=13,skip_cleanup=True
      

       
      It reproduces MB-41255 on 6.0.4-3082.

      It creates a 3 node cluster (172.23.106.89, 172.23.106.92 and 172.23.120.119).
      Create a default bucket with 2 replicas.
      Creates 10k docs with xattrs.
      Create another 350k docs to push the default bucket into heavy dgm(10%).
      Now deletes those 10k docs with delWithMeta.

      So the bucket default has 350000 active docs and 720000 replica docs.

      Now we try to upgrade the cluster to 6.0.4-MP2 node(by adding 172.23.120.120) and removing a 6.0.4 node(172.23.120.119) and starting swap rebalance. Rebalance hangs at 0%.

      cbcollect_info attached.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Daniel Owen - can you comment on the above logs?

            Balakumaran.Gopal Balakumaran Gopal added a comment - Daniel Owen - can you comment on the above logs?
            owend Daniel Owen added a comment -

            Hi Balakumaran Gopal, I believe the following WARNING message is OK.

            2020-09-18T05:11:36.496631-07:00 WARNING (default) Notified the timeout on seqno for vbucket 337, Check for: 314, Persisted upto: 309, cookie 0x7f19fb6a3b60
            

            as we see it is ultimately persisted.

            2020-09-18T07:12:30.717947-07:00 WARNING (default) Notified the timeout on seqno for vbucket 337, Check for: 314, Persisted upto: 309, cookie 0x7f19fb6a3b60
            2020-09-18T07:12:49.992647-07:00 INFO (default) Notified the completion of seqno for vbucket 337, Check for: 314, Persisted upto: 315, cookie 0x7f19fb6a3b60
            

            owend Daniel Owen added a comment - Hi Balakumaran Gopal , I believe the following WARNING message is OK. 2020-09-18T05:11:36.496631-07:00 WARNING (default) Notified the timeout on seqno for vbucket 337, Check for: 314, Persisted upto: 309, cookie 0x7f19fb6a3b60 as we see it is ultimately persisted. 2020-09-18T07:12:30.717947-07:00 WARNING (default) Notified the timeout on seqno for vbucket 337, Check for: 314, Persisted upto: 309, cookie 0x7f19fb6a3b60 2020-09-18T07:12:49.992647-07:00 INFO (default) Notified the completion of seqno for vbucket 337, Check for: 314, Persisted upto: 315, cookie 0x7f19fb6a3b60
            owend Daniel Owen added a comment -

            Hey Jim Walker. I did notice

            s3://cb-customers-secure/rebalance_hung_mp2_4/2020-09-21/collectinfo-2020-09-21t052255-ns_1@172.23.120.120.zip
            s3://cb-customers-secure/rebalance_hung_mp2_4/2020-09-21/collectinfo-2020-09-21t052255-ns_1@172.23.120.122.zip
            s3://cb-customers-secure/rebalance_hung_mp2_4/2020-09-21/collectinfo-2020-09-21t052255-ns_1@172.23.120.123.zip
            

            message of the following format End stream received with opaque "X" but expected "Y" e.g.

            End stream received with opaque 53 but expected 98
            

            You got any ideas why the opaque is not expected, and if we need to worry about it?

            thanks

            owend Daniel Owen added a comment - Hey Jim Walker . I did notice s3://cb-customers-secure/rebalance_hung_mp2_4/2020-09-21/collectinfo-2020-09-21t052255-ns_1@172.23.120.120.zip s3://cb-customers-secure/rebalance_hung_mp2_4/2020-09-21/collectinfo-2020-09-21t052255-ns_1@172.23.120.122.zip s3://cb-customers-secure/rebalance_hung_mp2_4/2020-09-21/collectinfo-2020-09-21t052255-ns_1@172.23.120.123.zip message of the following format End stream received with opaque "X" but expected "Y" e.g. End stream received with opaque 53 but expected 98 You got any ideas why the opaque is not expected, and if we need to worry about it? thanks
            jwalker Jim Walker added a comment - - edited

            Daniel Owen this is an issue seen before (in numerous logs) and it was finally tracked down and fixed in 6.5 - see MB-34951

            jwalker Jim Walker added a comment - - edited Daniel Owen this is an issue seen before (in numerous logs) and it was finally tracked down and fixed in 6.5 - see MB-34951
            owend Daniel Owen added a comment -

            Thanks Jim Walker as the patch is to change the log message an return ENGINE_SUCCESS instead of ENGINE_KEY_ENOENT, I'm OK that the message is not of concern and hence we can resolve the MB.

            owend Daniel Owen added a comment - Thanks Jim Walker as the patch is to change the log message an return ENGINE_SUCCESS instead of ENGINE_KEY_ENOENT, I'm OK that the message is not of concern and hence we can resolve the MB.

            People

              Balakumaran.Gopal Balakumaran Gopal
              Balakumaran.Gopal Balakumaran Gopal
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty