Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-5651

modify ec2 security group and open traffic to/from port 11209(Replication doesn't work on EC2.)

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Security Level: Public
    • Environment:
      EC2, Ubuntu 12.04

      ec2-174-129-82-112.compute-1.amazonaws.com (Administrator:password, root:couchbase)

      Description

      Build:
      – 2.0.0-1351-rel

      Security group:
      – couchbase

      Steps:
      – create 4 nodes cluster (nodes use private ip addresses)
      – create default bucket with 1 replica enabled
      – insert arbitrary number of items (2M in my case)

      Actual result:
      – no doc is replicated

      Notice:
      – It's not about XDCR.

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        pavelpaulau Pavel Paulau created issue -
        Hide
        chiyoung Chiyoung Seo added a comment -

        Mike, please do the initial investigation. Thanks.

        Show
        chiyoung Chiyoung Seo added a comment - Mike, please do the initial investigation. Thanks.
        chiyoung Chiyoung Seo made changes -
        Field Original Value New Value
        Assignee Chiyoung Seo [ chiyoung ] Mike Wiederhold [ mikew ]
        Hide
        mikew Mike Wiederhold added a comment -

        Ebucketmigrator is timing out while connecting to memcached. It looks like memcached might have crashed or become unresponsive after the initial vbucket creation mechanism. This appears to be the case since no longer see stats updates from ns_server in the logs. There are no crash reports from memcached though.

        =========================CRASH REPORT=========================
        crasher:
        initial call: ebucketmigrator_srv:init/1
        pid: <0.3854.0>
        registered_name: []
        exception error: no match of right hand side value

        {error,timeout}

        in function ebucketmigrator_srv:connect/4
        in call from ebucketmigrator_srv:init/1
        ancestors: ['ns_vbm_new_sup-default','ns_memcached_sup-default',
        'single_bucket_sup-default',<0.1187.0>]
        messages: []
        links: Port<0.8470>,<0.1200.0>
        dictionary: []
        trap_exit: false
        status: running
        heap_size: 610
        stack_size: 24
        reductions: 7063
        neighbours:

        [error_logger:error] [2012-06-22 16:46:33] [ns_1@10.121.13.111:error_logger:ale_error_logger_handler:log_report:72]

        Show
        mikew Mike Wiederhold added a comment - Ebucketmigrator is timing out while connecting to memcached. It looks like memcached might have crashed or become unresponsive after the initial vbucket creation mechanism. This appears to be the case since no longer see stats updates from ns_server in the logs. There are no crash reports from memcached though. =========================CRASH REPORT========================= crasher: initial call: ebucketmigrator_srv:init/1 pid: <0.3854.0> registered_name: [] exception error: no match of right hand side value {error,timeout} in function ebucketmigrator_srv:connect/4 in call from ebucketmigrator_srv:init/1 ancestors: ['ns_vbm_new_sup-default','ns_memcached_sup-default', 'single_bucket_sup-default',<0.1187.0>] messages: [] links: Port<0.8470>,<0.1200.0> dictionary: [] trap_exit: false status: running heap_size: 610 stack_size: 24 reductions: 7063 neighbours: [error_logger:error] [2012-06-22 16:46:33] [ns_1@10.121.13.111:error_logger:ale_error_logger_handler:log_report:72]
        mikew Mike Wiederhold made changes -
        Priority Blocker [ 1 ] Major [ 3 ]
        Hide
        mikew Mike Wiederhold added a comment -

        Lowering priority since I don't have much information to address further.

        Show
        mikew Mike Wiederhold added a comment - Lowering priority since I don't have much information to address further.
        Hide
        mikew Mike Wiederhold added a comment -

        Assigning back to Pavel. Is this reproducible or do you have a cluster I can look at?

        Show
        mikew Mike Wiederhold added a comment - Assigning back to Pavel. Is this reproducible or do you have a cluster I can look at?
        mikew Mike Wiederhold made changes -
        Assignee Mike Wiederhold [ mikew ] Pavel Paulau [ pavelpaulau ]
        Hide
        karan Karan Kumar (Inactive) added a comment -

        @Pavel:

        Please put the build number on the ticket.

        Show
        karan Karan Kumar (Inactive) added a comment - @Pavel: Please put the build number on the ticket.
        Hide
        pavelpaulau Pavel Paulau added a comment -

        try this cluster:

        ec2-174-129-98-20.compute-1.amazonaws.com
        ec2-23-20-111-39.compute-1.amazonaws.com
        ec2-23-22-235-210.compute-1.amazonaws.com
        ec2-107-20-81-224.compute-1.amazonaws.com

        Show
        pavelpaulau Pavel Paulau added a comment - try this cluster: ec2-174-129-98-20.compute-1.amazonaws.com ec2-23-20-111-39.compute-1.amazonaws.com ec2-23-22-235-210.compute-1.amazonaws.com ec2-107-20-81-224.compute-1.amazonaws.com
        pavelpaulau Pavel Paulau made changes -
        Assignee Pavel Paulau [ pavelpaulau ] Mike Wiederhold [ mikew ]
        pavelpaulau Pavel Paulau made changes -
        Description Steps:
        -- create 4 nodes cluster (nodes use private ip addresses)
        -- create default bucket with 1 replica enabled
        -- insert arbitrary number of items (2M in my case)

        Actual result:
        -- no doc is replicated

        Notice:
        -- It's not about XDCR.
        Build:
        2.0.0-1351-rel

        Steps:
        -- create 4 nodes cluster (nodes use private ip addresses)
        -- create default bucket with 1 replica enabled
        -- insert arbitrary number of items (2M in my case)

        Actual result:
        -- no doc is replicated

        Notice:
        -- It's not about XDCR.
        pavelpaulau Pavel Paulau made changes -
        Description Build:
        2.0.0-1351-rel

        Steps:
        -- create 4 nodes cluster (nodes use private ip addresses)
        -- create default bucket with 1 replica enabled
        -- insert arbitrary number of items (2M in my case)

        Actual result:
        -- no doc is replicated

        Notice:
        -- It's not about XDCR.
        Build:
        -- 2.0.0-1351-rel

        Security group:
        -- couchbase

        Steps:
        -- create 4 nodes cluster (nodes use private ip addresses)
        -- create default bucket with 1 replica enabled
        -- insert arbitrary number of items (2M in my case)

        Actual result:
        -- no doc is replicated

        Notice:
        -- It's not about XDCR.
        mikew Mike Wiederhold made changes -
        Component/s couchbase-bucket [ 10173 ]
        Hide
        mikew Mike Wiederhold added a comment -

        The security group doesn't have port 11209 open and this port is used by ns_server for replication. Karan has just edited the security group to enable this port.

        Show
        mikew Mike Wiederhold added a comment - The security group doesn't have port 11209 open and this port is used by ns_server for replication. Karan has just edited the security group to enable this port.
        mikew Mike Wiederhold made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Won't Fix [ 2 ]
        farshid Farshid Ghods (Inactive) made changes -
        Summary Replication doesn't work on EC2. modify ec2 security group and open traffic to/from port 11209(Replication doesn't work on EC2.)
        Labels 1.8.1-release-notes
        Hide
        pavelpaulau Pavel Paulau added a comment -
        Show
        pavelpaulau Pavel Paulau added a comment - I think we also should fix this: http://www.couchbase.com/docs/couchbase-manual-2.0/couchbase-network-ports.html
        Show
        farshid Farshid Ghods (Inactive) added a comment - add port 11209 to 2.0 and 1.8 manual http://www.couchbase.com/docs/couchbase-manual-2.0/couchbase-network-ports.html
        farshid Farshid Ghods (Inactive) made changes -
        Resolution Won't Fix [ 2 ]
        Status Resolved [ 5 ] Reopened [ 4 ]
        Assignee Mike Wiederhold [ mikew ] MC Brown [ mccouch ]
        farshid Farshid Ghods (Inactive) made changes -
        Component/s documentation [ 10012 ]
        mikew Mike Wiederhold made changes -
        Component/s couchbase-bucket [ 10173 ]
        Hide
        mccouch MC Brown (Inactive) added a comment -

        Documentation has been updated with port 11209

        Show
        mccouch MC Brown (Inactive) added a comment - Documentation has been updated with port 11209
        mccouch MC Brown (Inactive) made changes -
        Status Reopened [ 4 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        pavelpaulau Pavel Paulau made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            mccouch MC Brown (Inactive)
            Reporter:
            pavelpaulau Pavel Paulau
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes