Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-58654

[AWS]: Bucket is stuck in warmup immediately after creation.

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Test Blocker
    • 7.6.0
    • 7.6.0
    • couchbase-bucket, ns_server
    • toy-couchbase-cloud-server-7.6.0-1498-x86_64-304

    Description

      Note: This is the first toy AMI created with provisioned profile that's why the name toy otherwise its a regular build. cc: Ming Ho

      1. Create a 3 kv node and 2 gsi, 2 query node cluster.
      2. Create a magma bucket with 1024MB RAM.
      3. The bucket is stuck in warmup.

      memcached logs are flooded with:

      2023-09-13T23:35:27.450398+00:00 INFO 4: Client {"ip":"127.0.0.1","port":41410} authenticated as @ns_server. Mechanism:[PLAIN]
      2023-09-13T23:35:27.450483+00:00 INFO 4: HELO [ns_server:ns_memcached-default0] [ {"ip":"127.0.0.1","port":41410} - {"ip":"127.0.0.1","port":11209} (System, @ns_server) ]
      2023-09-13T23:35:27.458478+00:00 INFO 66: Client {"ip":"127.0.0.1","port":41436} authenticated as @ns_server. Mechanism:[PLAIN]
      2023-09-13T23:35:27.458557+00:00 INFO 66: HELO [ns_server:ns_memcached_sockets_pool] [ {"ip":"127.0.0.1","port":41436} - {"ip":"127.0.0.1","port":11209} (System, @ns_server) ]
      2023-09-13T23:35:27.468127+00:00 INFO 4: Client {"ip":"127.0.0.1","port":41438} authenticated as @ns_server. Mechanism:[PLAIN]
      2023-09-13T23:35:27.468212+00:00 INFO 4: HELO [ns_server:ns_memcached-default0] [ {"ip":"127.0.0.1","port":41438} - {"ip":"127.0.0.1","port":11209} (System, @ns_server) ]
      2023-09-13T23:35:27.476019+00:00 INFO 66: Client {"ip":"127.0.0.1","port":41460} authenticated as @ns_server. Mechanism:[PLAIN]
      2023-09-13T23:35:27.476105+00:00 INFO 66: HELO [ns_server:ns_memcached_sockets_pool] [ {"ip":"127.0.0.1","port":41460} - {"ip":"127.0.0.1","port":11209} (System, @ns_server) ]
      

      From debug logs:

      ========================CRASH REPORT=========================
        crasher:
          initial call: janitor_agent:init/1
          pid: <0.13426.17>
          registered_name: 'janitor_agent-default0'
          exception exit: {{badarg,
                               [{ejson,reverse_tokens,[undefined],[]},
                                {ejson,nif_decode,1,
                                    [{file,
                                         "/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/ejson/ejson.erl"},
                                     {line,40}]},
                                {ns_memcached,'-get_bucket_stats/3-fun-1-',4,
                                    [{file,"src/ns_memcached.erl"},{line,1806}]},
                                {ns_memcached,'-perform_very_long_call/3-fun-0-',2,
                                    [{file,"src/ns_memcached.erl"},{line,404}]},
                                {ns_memcached_sockets_pool,
                                    '-executing_on_socket/3-fun-0-',3,
                                    [{file,"src/ns_memcached_sockets_pool.erl"},
                                     {line,88}]},
                                {async,'-async_init/4-fun-1-',3,
                                    [{file,"src/async.erl"},{line,199}]}]},
                           {gen_server,call,
                               [{'ns_memcached-default0',
                                    'ns_1@svc-d-node-001.cckggpujdpy-qv9m.sandbox.nonprod-project-avengers.com'},
                                {raw_stats,<<"dcp">>,
                                    #Fun<dcp_replicator.3.103011606>,[]},
                                300000]}}
            in function  gen_server:call/3 (gen_server.erl, line 385)
            in call from ns_memcached:do_call/4 (src/ns_memcached.erl, line 1624)
            in call from dcp_replicator:get_connections/1 (src/dcp_replicator.erl, line 296)
            in call from dcp_sup:nuke/1 (src/dcp_sup.erl, line 114)
            in call from janitor_agent:init/1 (src/janitor_agent.erl, line 591)
          ancestors: ['janitor_agent_sup-default0',
                        'single_bucket_kv_sup-default0',ns_bucket_sup,
                        ns_bucket_worker_sup,ns_server_sup,ns_server_nodes_sup,
                        <0.2902.0>,ns_server_cluster_sup,root_sup,<0.154.0>]
          message_queue_len: 0
          messages: []
          links: [<0.13297.17>]
          dictionary: []
          trap_exit: false
          status: running
          heap_size: 6772
          stack_size: 28
          reductions: 3092
        neighbours:
      

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              ritesh.agarwal Ritesh Agarwal
              ritesh.agarwal Ritesh Agarwal
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty