Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-5651

modify ec2 security group and open traffic to/from port 11209(Replication doesn't work on EC2.)

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Security Level: Public
    • Environment:
      EC2, Ubuntu 12.04

      ec2-174-129-82-112.compute-1.amazonaws.com (Administrator:password, root:couchbase)

      Description

      Build:
      – 2.0.0-1351-rel

      Security group:
      – couchbase

      Steps:
      – create 4 nodes cluster (nodes use private ip addresses)
      – create default bucket with 1 replica enabled
      – insert arbitrary number of items (2M in my case)

      Actual result:
      – no doc is replicated

      Notice:
      – It's not about XDCR.

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        chiyoung Chiyoung Seo added a comment -

        Mike, please do the initial investigation. Thanks.

        Show
        chiyoung Chiyoung Seo added a comment - Mike, please do the initial investigation. Thanks.
        Hide
        mikew Mike Wiederhold added a comment -

        Ebucketmigrator is timing out while connecting to memcached. It looks like memcached might have crashed or become unresponsive after the initial vbucket creation mechanism. This appears to be the case since no longer see stats updates from ns_server in the logs. There are no crash reports from memcached though.

        =========================CRASH REPORT=========================
        crasher:
        initial call: ebucketmigrator_srv:init/1
        pid: <0.3854.0>
        registered_name: []
        exception error: no match of right hand side value

        {error,timeout}

        in function ebucketmigrator_srv:connect/4
        in call from ebucketmigrator_srv:init/1
        ancestors: ['ns_vbm_new_sup-default','ns_memcached_sup-default',
        'single_bucket_sup-default',<0.1187.0>]
        messages: []
        links: Port<0.8470>,<0.1200.0>
        dictionary: []
        trap_exit: false
        status: running
        heap_size: 610
        stack_size: 24
        reductions: 7063
        neighbours:

        [error_logger:error] [2012-06-22 16:46:33] [ns_1@10.121.13.111:error_logger:ale_error_logger_handler:log_report:72]

        Show
        mikew Mike Wiederhold added a comment - Ebucketmigrator is timing out while connecting to memcached. It looks like memcached might have crashed or become unresponsive after the initial vbucket creation mechanism. This appears to be the case since no longer see stats updates from ns_server in the logs. There are no crash reports from memcached though. =========================CRASH REPORT========================= crasher: initial call: ebucketmigrator_srv:init/1 pid: <0.3854.0> registered_name: [] exception error: no match of right hand side value {error,timeout} in function ebucketmigrator_srv:connect/4 in call from ebucketmigrator_srv:init/1 ancestors: ['ns_vbm_new_sup-default','ns_memcached_sup-default', 'single_bucket_sup-default',<0.1187.0>] messages: [] links: Port<0.8470>,<0.1200.0> dictionary: [] trap_exit: false status: running heap_size: 610 stack_size: 24 reductions: 7063 neighbours: [error_logger:error] [2012-06-22 16:46:33] [ns_1@10.121.13.111:error_logger:ale_error_logger_handler:log_report:72]
        Hide
        mikew Mike Wiederhold added a comment -

        Lowering priority since I don't have much information to address further.

        Show
        mikew Mike Wiederhold added a comment - Lowering priority since I don't have much information to address further.
        Hide
        mikew Mike Wiederhold added a comment -

        Assigning back to Pavel. Is this reproducible or do you have a cluster I can look at?

        Show
        mikew Mike Wiederhold added a comment - Assigning back to Pavel. Is this reproducible or do you have a cluster I can look at?
        Hide
        karan Karan Kumar (Inactive) added a comment -

        @Pavel:

        Please put the build number on the ticket.

        Show
        karan Karan Kumar (Inactive) added a comment - @Pavel: Please put the build number on the ticket.
        Hide
        pavelpaulau Pavel Paulau added a comment -

        try this cluster:

        ec2-174-129-98-20.compute-1.amazonaws.com
        ec2-23-20-111-39.compute-1.amazonaws.com
        ec2-23-22-235-210.compute-1.amazonaws.com
        ec2-107-20-81-224.compute-1.amazonaws.com

        Show
        pavelpaulau Pavel Paulau added a comment - try this cluster: ec2-174-129-98-20.compute-1.amazonaws.com ec2-23-20-111-39.compute-1.amazonaws.com ec2-23-22-235-210.compute-1.amazonaws.com ec2-107-20-81-224.compute-1.amazonaws.com
        Hide
        mikew Mike Wiederhold added a comment -

        The security group doesn't have port 11209 open and this port is used by ns_server for replication. Karan has just edited the security group to enable this port.

        Show
        mikew Mike Wiederhold added a comment - The security group doesn't have port 11209 open and this port is used by ns_server for replication. Karan has just edited the security group to enable this port.
        Hide
        pavelpaulau Pavel Paulau added a comment -
        Show
        pavelpaulau Pavel Paulau added a comment - I think we also should fix this: http://www.couchbase.com/docs/couchbase-manual-2.0/couchbase-network-ports.html
        Show
        farshid Farshid Ghods (Inactive) added a comment - add port 11209 to 2.0 and 1.8 manual http://www.couchbase.com/docs/couchbase-manual-2.0/couchbase-network-ports.html
        Hide
        mccouch MC Brown (Inactive) added a comment -

        Documentation has been updated with port 11209

        Show
        mccouch MC Brown (Inactive) added a comment - Documentation has been updated with port 11209

          People

          • Assignee:
            mccouch MC Brown (Inactive)
            Reporter:
            pavelpaulau Pavel Paulau
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes