Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-32081

memcached marks a valid set_with_meta with options and extra_len as invalid

    XMLWordPrintable

Details

    • Untriaged
    • Yes

    Description

      cbbackupmgr is failing with below error message in backup.log since 6.5.0-1645 build 

       

      2018-11-19T12:18:32.371-08:00 ERRO: invalid arguments – plan.(*data).execute() at data.go:90

      2018-11-19T12:18:32.371-08:00 (Plan) Transfer plan failed due to error invalid arguments

      2018-11-19T12:18:32.371-08:00 (Cmd) Error restoring cluster: invalid arguments

       

      Attachments

        1. backup.tar.gz
          3.44 MB
        2. bad.pcap
          1.29 MB
        3. good.pcap
          4 kB
        4. tcpdump-request.png
          tcpdump-request.png
          105 kB
        5. tcpdump-response.png
          tcpdump-response.png
          124 kB

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            pvarley Patrick Varley added a comment - - edited

            From a tcpdump Couchbase Server is saying the set_with_meta had no value (or is invalid, I need to check if wireshark assumes invalid is no value):

            From the requests does have the value:

            pvarley Patrick Varley added a comment - - edited From a tcpdump Couchbase Server is saying the set_with_meta had no value (or is invalid, I need to check if wireshark assumes invalid is no value): From the requests does have the value:

            I think this is a memcached problem:

            1. Given that Carlos Gonzalez Betancort has tested this version of cbbackupmgr against Couchbase Server 6.0.0 and it work
            2. The tcpdump screenshot provided above
            3. The fact that there has been kv_engine changes in this part of the code base.

            I will see if I produce a unit test out side of cbbackupmgr to backup this theory.

            Ritam Sharma and Sharath Sulochana do we have functional test for the underlying protocol? We hit a few issues in the past with set_with_meta and delete_with_meta operations.

            pvarley Patrick Varley added a comment - I think this is a memcached problem: Given that Carlos Gonzalez Betancort has tested this version of cbbackupmgr against Couchbase Server 6.0.0 and it work The tcpdump screenshot provided above The fact that there has been kv_engine changes in this part of the code base. I will see if I produce a unit test out side of cbbackupmgr to backup this theory. Ritam Sharma and Sharath Sulochana do we have functional test for the underlying protocol? We hit a few issues in the past with set_with_meta and delete_with_meta operations.
            pvarley Patrick Varley added a comment - - edited

            Created a unit test setwithMeta validator is broken when options and meta len is set, which is valid according to the spec

            Unit test using the cb_bin_client

            #!/usr/bin/env python
             
            from __future__ import print_function
             
            from cb_bin_client import MemcachedClient
            from cb_bin_client import MemcachedError
             
            import sys
             
            def main(args):
             
                serverip = 'localhost'
                port = '11210'
                bucket = 'gamesim-sample'
                user = 'Administrator'
                password = 'password'
             
                key = 'test'
                set_exp = 0
                set_flags = 0x02000000
                set_value = '{"set_with_meta":"test"}'
             
                client = MemcachedClient(serverip, port)
                client.sasl_auth_plain(user, password)
                client.bucket_select(bucket)
                client.set(key, set_exp, set_flags, set_value)
                deleted, flags, exp, seqno, remote_cas = client.getMeta(key)
                print('deleted: {}, flags {}, exp {}, seqno {}, remote_cas {}'.format(deleted, flags, exp, seqno, remote_cas))
             
                flags2, remote_case2, value  = client.get(key)
                print("flags2: {}, remote_case2: {}, value: {}".format(flags2, remote_case2, value))
                for x in xrange(1):
                    try:
                        set_with_meta = client.setWithMeta(key, value, exp, flags, seqno, remote_cas, 0, 0)
                        print(meta_with_meta)
             
                    except MemcachedError as error:
                        print ("key '{}' error: {}".format(key, error))
             
            if __name__ == '__main__':
                sys.exit(main(sys.argv))
            

            I had to patch two functions in cb_bin_client.py to add in support for options and meta_len:

                def _doMetaCmd(self, cmd, key, value, cas, exp, flags, seqno, remote_cas, options, meta_len):
                    extra = struct.pack('>IIQQIH', flags, exp, seqno, remote_cas, options, meta_len)
                    return self._doCmd(cmd, key, value, extra, cas)
             
                def setWithMeta(self, key, value, exp, flags, seqno, remote_cas, options, meta_len):
                    """Set a value and its meta data in the memcached server."""
                    return self._doMetaCmd(couchbaseConstants.CMD_SET_WITH_META,
                                           key, value, 0, exp, flags, seqno, remote_cas, options, meta_len)
            

            I will do a good patch for cb_bin_client later as options and meta_len is fully optional.

            I have tested this against Couchbase Server 6.0.0 and it works:

            [vagrant@node1-mad-hatter-testing-centos7 python]$ ./getMeta.py
            deleted: 0, flags 33554432, exp 0, seqno 39, remote_cas 1542845114800799744
            flags2: 33554432, remote_case2: 1542845114800799744, value: {"set_with_meta":"test"}
            key 'test' error: Memcached error #4
            #Against Couchbase Server 6.0.0 GA
            [vagrant@node1-mad-hatter-testing-centos7 python]$ ./getMeta.py
            deleted: 0, flags 33554432, exp 0, seqno 6, remote_cas 1542845066266411008
            flags2: 33554432, remote_case2: 1542845066266411008, value: {"set_with_meta":"test"}
            key 'test' error: Memcached error #2
            

            pvarley Patrick Varley added a comment - - edited Created a unit test setwithMeta validator is broken when options and meta len is set, which is valid according to the spec Unit test using the cb_bin_client #!/usr/bin/env python   from __future__ import print_function   from cb_bin_client import MemcachedClient from cb_bin_client import MemcachedError   import sys   def main(args):   serverip = 'localhost' port = '11210' bucket = 'gamesim-sample' user = 'Administrator' password = 'password'   key = 'test' set_exp = 0 set_flags = 0x02000000 set_value = '{"set_with_meta":"test"}'   client = MemcachedClient(serverip, port) client.sasl_auth_plain(user, password) client.bucket_select(bucket) client. set (key, set_exp, set_flags, set_value) deleted, flags, exp, seqno, remote_cas = client.getMeta(key) print ( 'deleted: {}, flags {}, exp {}, seqno {}, remote_cas {}' . format (deleted, flags, exp, seqno, remote_cas))   flags2, remote_case2, value = client.get(key) print ( "flags2: {}, remote_case2: {}, value: {}" . format (flags2, remote_case2, value)) for x in xrange ( 1 ): try : set_with_meta = client.setWithMeta(key, value, exp, flags, seqno, remote_cas, 0 , 0 ) print (meta_with_meta)   except MemcachedError as error: print ( "key '{}' error: {}" . format (key, error))   if __name__ = = '__main__' : sys.exit(main(sys.argv)) I had to patch two functions in cb_bin_client.py to add in support for options and meta_len: def _doMetaCmd( self , cmd, key, value, cas, exp, flags, seqno, remote_cas, options, meta_len): extra = struct.pack( '>IIQQIH' , flags, exp, seqno, remote_cas, options, meta_len) return self ._doCmd(cmd, key, value, extra, cas)   def setWithMeta( self , key, value, exp, flags, seqno, remote_cas, options, meta_len): """Set a value and its meta data in the memcached server.""" return self ._doMetaCmd(couchbaseConstants.CMD_SET_WITH_META, key, value, 0 , exp, flags, seqno, remote_cas, options, meta_len) I will do a good patch for cb_bin_client later as options and meta_len is fully optional. I have tested this against Couchbase Server 6.0.0 and it works: [vagrant@node1-mad-hatter-testing-centos7 python]$ ./getMeta.py deleted: 0, flags 33554432, exp 0, seqno 39, remote_cas 1542845114800799744 flags2: 33554432, remote_case2: 1542845114800799744, value: {"set_with_meta":"test"} key 'test' error: Memcached error #4 #Against Couchbase Server 6.0.0 GA [vagrant@node1-mad-hatter-testing-centos7 python]$ ./getMeta.py deleted: 0, flags 33554432, exp 0, seqno 6, remote_cas 1542845066266411008 flags2: 33554432, remote_case2: 1542845066266411008, value: {"set_with_meta":"test"} key 'test' error: Memcached error #2

            Adding the tcpdump against Mad-hatter(bad.pcap) and 6.0.0 (good.pcap)

            pvarley Patrick Varley added a comment - Adding the tcpdump against Mad-hatter( bad.pcap ) and 6.0.0 ( good.pcap )

            Build couchbase-server-6.5.0-1704 contains kv_engine commit 4e9c492 with commit message:
            MB-32081, MB-32114: Format failures in *WithMeta

            build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.0-1704 contains kv_engine commit 4e9c492 with commit message: MB-32081 , MB-32114 : Format failures in *WithMeta

            People

              sharath.sulochana Sharath Sulochana (Inactive)
              sharath.sulochana Sharath Sulochana (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty