Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-6057

unable to create a bucket under windows

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0-beta
    • Component/s: couchbase-bucket
    • Security Level: Public
    • Labels:
      None
    • Environment:
      windows server r2 2008 64-bit

      Description

      Install couchbase server 2.0.0-1516 on windows server 2008 64 bit (1 node).
      Run smoke test which create and delete several time default and non default bucket.
      At tests create multi none default buckets, the test will fail due to unable to create bucket.
      -t setgettests.MembaseBucket.test_value_100b
      -t setgettests.MembaseBucket.test_value_500kb

      On the UI the node's and bucket's statuses are pending.
      CRASH REPORTS appear in logs:

      =========================CRASH REPORT=========================
      crasher:
      initial call: supervisor_cushion:init/1
      pid: <0.23646.8>
      registered_name: []
      exception exit:

      {abnormal,255}

      in function gen_server:terminate/6
      ancestors: [ns_port_sup,ns_server_sup,ns_server_cluster_sup,<0.67.0>]
      messages: []=========================CRASH REPORT=========================
      crasher:
      initial call: ns_memcached:init/1
      pid: <0.23611.8>
      registered_name: 'ns_memcached-default'
      exception exit: {badmatch,{error,closed}}
      in function gen_server:terminate/6
      ancestors: ['ns_memcached_sup-default','single_bucket_sup-default',
      <0.22907.8>]
      messages: []
      links: [<0.23665.8>,<0.23666.8>,<0.23668.8>,<0.64.0>,<0.22928.8>]
      dictionary: []
      trap_exit: true
      status: running
      heap_size: 17711
      stack_size: 24
      reductions: 17835
      neighbours:
      neighbour: [

      {pid,<0.23668.8>},
      {registered_name,[]},
      {initial_call,{erlang,apply,['Argument__1','Argument__2']}},
      {current_function,{gen,do_call,4}},
      {ancestors,['ns_memcached-default', 'ns_memcached_sup-default', 'single_bucket_sup-default',<0.22907.8>]},
      {messages,[]},
      {links,[<0.23611.8>,#Port<0.142256>]},
      {dictionary,[]},
      {trap_exit,false},
      {status,waiting},
      {heap_size,10946},
      {stack_size,24},
      {reductions,1151}]
      links: [<0.407.0>]
      dictionary: []
      trap_exit: true
      status: running
      heap_size: 1597
      stack_size: 24
      reductions: 2361
      neighbours:

      =========================CRASH REPORT=========================
      crasher:
      initial call: ns_memcached:init/1
      pid: <0.23611.8>
      registered_name: 'ns_memcached-default'
      exception exit: {badmatch,{error,closed}}
      in function gen_server:terminate/6
      ancestors: ['ns_memcached_sup-default','single_bucket_sup-default',
      <0.22907.8>]
      messages: []
      links: [<0.23665.8>,<0.23666.8>,<0.23668.8>,<0.64.0>,<0.22928.8>]
      dictionary: []
      trap_exit: true
      status: running
      heap_size: 17711
      stack_size: 24
      reductions: 17835
      neighbours:
      neighbour: [{pid,<0.23668.8>}

      ,

      {registered_name,[]}

      ,
      {initial_call,{erlang,apply,['Argument__1','Argument__2']}},
      {current_function,{gen,do_call,4}},

      {ancestors,['ns_memcached-default', 'ns_memcached_sup-default', 'single_bucket_sup-default',<0.22907.8>]}

      ,

      {messages,[]}

      ,

      {links,[<0.23611.8>,#Port<0.142256>]}

      ,

      {dictionary,[]}

      ,

      {trap_exit,false}

      ,

      {status,waiting}

      ,

      {heap_size,10946}

      ,

      {stack_size,24}

      ,

      {reductions,1151}

      ]

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        iryna iryna created issue -
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        Memcached simply crashed:

        [user:info] [2012-07-30 14:56:27] [ns_1@10.3.2.188:ns_port_memcached:ns_port_server:handle_info:107] Port server memcached on node 'ns_1@10.3.2.188' exited with status 255. Restarting. Messages: Trying to connect to mccouch: "localhost:11213"
        Connected to mccouch: "localhost:11213"
        Warning: failed to load the engine session stats due to IO exception "basic_ios::clear"
        Failed to load mutation log, falling back to key dump
        Extension support isn't implemented in this version of bucket_engine
        metadata loaded in 2332 usec
        [ns_server:info] [2012-07-30 14:56:27] [ns_1@10.3.2.188:ns_port_memcached:ns_port_server:log:169] memcached<0.7213.9>: Connected to mccouch: "localhost:11213"
        memcached<0.7213.9>: Warning: failed to load the engine session stats due to IO exception "basic_ios::clear"
        memcached<0.7213.9>: Failed to load mutation log, falling back to key dump
        memcached<0.7213.9>: Extension support isn't implemented in this version of bucket_engine
        memcached<0.7213.9>: metadata loaded in 2332 usec

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - Memcached simply crashed: [user:info] [2012-07-30 14:56:27] [ns_1@10.3.2.188:ns_port_memcached:ns_port_server:handle_info:107] Port server memcached on node 'ns_1@10.3.2.188' exited with status 255. Restarting. Messages: Trying to connect to mccouch: "localhost:11213" Connected to mccouch: "localhost:11213" Warning: failed to load the engine session stats due to IO exception "basic_ios::clear" Failed to load mutation log, falling back to key dump Extension support isn't implemented in this version of bucket_engine metadata loaded in 2332 usec [ns_server:info] [2012-07-30 14:56:27] [ns_1@10.3.2.188:ns_port_memcached:ns_port_server:log:169] memcached<0.7213.9>: Connected to mccouch: "localhost:11213" memcached<0.7213.9>: Warning: failed to load the engine session stats due to IO exception "basic_ios::clear" memcached<0.7213.9>: Failed to load mutation log, falling back to key dump memcached<0.7213.9>: Extension support isn't implemented in this version of bucket_engine memcached<0.7213.9>: metadata loaded in 2332 usec
        alkondratenko Aleksey Kondratenko (Inactive) made changes -
        Field Original Value New Value
        Assignee Aleksey Kondratenko [ alkondratenko ] Peter Wansch [ peter ]
        Hide
        peter peter added a comment -

        Trond, can you take a look?

        Show
        peter peter added a comment - Trond, can you take a look?
        peter peter made changes -
        Assignee Peter Wansch [ peter ] Trond Norbye [ trond ]
        Fix Version/s 2.0-beta [ 10113 ]
        Sprint Status Current Sprint
        Component/s bucket-engine [ 10010 ]
        Hide
        trond Trond Norbye added a comment -

        From the attached logfile it seems to crash 4 seconds after the node was started. Is it possible to even start the build on windows? Can I get access to a server with the windows version and the failed version to do basic testing?

        Show
        trond Trond Norbye added a comment - From the attached logfile it seems to crash 4 seconds after the node was started. Is it possible to even start the build on windows? Can I get access to a server with the windows version and the failed version to do basic testing?
        trond Trond Norbye made changes -
        Assignee Trond Norbye [ trond ] Iryna Mironava [ iryna ]
        Hide
        iryna iryna added a comment -

        Win 7 64 bit VM: http://10.3.2.188:8091 with build #1516
        Administrator/password

        Show
        iryna iryna added a comment - Win 7 64 bit VM: http://10.3.2.188:8091 with build #1516 Administrator/password
        iryna iryna made changes -
        Assignee Iryna Mironava [ iryna ] Trond Norbye [ trond ]
        farshid Farshid Ghods (Inactive) made changes -
        Labels trunk-green-blockers
        Assignee Trond Norbye [ trond ] Jin Lim [ jin ]
        Hide
        farshid Farshid Ghods (Inactive) added a comment -

        Jin is also working on this - he is seeing that couchbase service does not start

        Show
        farshid Farshid Ghods (Inactive) added a comment - Jin is also working on this - he is seeing that couchbase service does not start
        chiyoung Chiyoung Seo made changes -
        Component/s couchbase-bucket [ 10173 ]
        Component/s bucket-engine [ 10010 ]
        Hide
        jin Jin Lim (Inactive) added a comment - - edited

        I was able to reproduce this locally and fixed it by applying the two reverts below.

        http://review.couchbase.org/#change,18972
        http://review.couchbase.org/#change,18973

        However, while as successfully starting stand-alone memcached with couchNotifier (aka mccouch) being disabled via command line and doing some I/Os (upto 1M) successfully. I still cannot start ns_server (erl process) via 'net start couchbaseserver". We need to have Bin merge the above reverts asap and continue debug from there. Probably we need erl engineers take a look at it as well.

        Show
        jin Jin Lim (Inactive) added a comment - - edited I was able to reproduce this locally and fixed it by applying the two reverts below. http://review.couchbase.org/#change,18972 http://review.couchbase.org/#change,18973 However, while as successfully starting stand-alone memcached with couchNotifier (aka mccouch) being disabled via command line and doing some I/Os (upto 1M) successfully. I still cannot start ns_server (erl process) via 'net start couchbaseserver". We need to have Bin merge the above reverts asap and continue debug from there. Probably we need erl engineers take a look at it as well.
        Hide
        jin Jin Lim (Inactive) added a comment -

        Downloaded the latest build and src code from the main trunk.

        "net start couchbaseserver" still fails starting memcached. The log indicates that memcached exited right after the warmup.
        However, I was able to start stand-alone memcached with the same ep.so library and finished running some I/Os. See the compiled output below:

        c:\Program Files\Couchbase\Server\bin>memcached -p 11211 -E ..\lib\memcached\ep.
        so -r -e "ht_size=12289;ht_locks=23;min_data_age=1;dbname=..\var\lib\couchbase\d
        ata\default;queue_age_cap=5;max_size=84901888;vb0=true;backend=couchdb" -u nobody
        Warning: failed to load the engine session stats due to IO exception "basic_ios
        ::clear" -> This is a normal error output for no stats.json file being found during the initial startup!
        Failed to load mutation log, falling back to key dump
        metadata loaded in 2749 usec
        warmup completed in 3601 usec

        Jins-MacBook-Pro:bin jin$ ./memcachetest -h 10.3.121.142:11211 -i 100 -V
        Average with 1 threads
        Get operations:
        #of ops. min max avg max90th max95th max99th
        6716 1504 us 4555 ms 22 ms 40 ms 54 ms 110 ms

        Set operations:
        #of ops. min max avg max90th max95th max99th
        3284 1699 us 937 ms 25 ms 46 ms 60 ms 151 ms

        Usr: 0.238064
        Sys: 1.037286
        Tot: 236.509054
        Server time:
        Usr: 0.000000
        Sys: 0.000000
        Total gets: 0
        Total sets: 100
        Jins-MacBook-Pro:bin jin$ ./cbstats 10.3.121.142:11211 timings
        data_age (1682 total)
        1s - 2s : ( 74.97%) 1261 #######################################################
        2s - 4s : (100.00%) 421 ##################
        disk_commit (143909 total)
        0 - 1s : (100.00%) 143909 ########################################################################
        disk_insert (100 total)
        8us - 16us : ( 31.00%) 31 #######################
        16us - 32us : ( 70.00%) 39 #############################
        32us - 64us : ( 98.00%) 28 #####################
        64us - 128us : ( 99.00%) 1
        256us - 512us : (100.00%) 1
        disk_update (1582 total)
        0 - 1us : ( 0.06%) 1
        8us - 16us : ( 18.27%) 288 #############
        16us - 32us : ( 58.03%) 629 #############################
        32us - 64us : ( 99.87%) 662 ###############################
        64us - 128us : (100.00%) 2
        disk_vbstate_snapshot (1 total)
        1ms - 2ms : (100.00%) 1 #############################################################################
        get_cmd (6716 total)
        0 - 1us : ( 0.07%) 5
        8us - 16us : ( 33.41%) 2239 ########################
        16us - 32us : ( 97.56%) 4308 ###############################################
        32us - 64us : ( 99.82%) 152 #
        64us - 128us : ( 99.99%) 11
        128us - 256us : (100.00%) 1
        get_stats_cmd (13 total)
        64us - 128us : ( 7.69%) 1 #####
        128us - 256us : ( 46.15%) 5 #############################
        1ms - 2ms : ( 53.85%) 1 #####
        2ms - 4ms : (100.00%) 6 ###################################
        item_alloc_sizes (3384 total)
        32 - 64 : ( 0.89%) 30
        64 - 128 : ( 4.05%) 107 ##
        128 - 256 : ( 7.36%) 112 ##
        256 - 512 : ( 13.09%) 194 ####
        512 - 1KB : ( 28.99%) 538 ###########
        1KB - 2KB : ( 50.35%) 723 ###############
        2KB - 4KB : (100.00%) 1680 ####################################
        storage_age (1682 total)
        0 - 1s : ( 74.32%) 1250 ######################################################
        1s - 2s : (100.00%) 432 ###################
        Jins-MacBook-Pro:bin jin$ ./cbstats 10.3.121.142:11211 all | grep curr
        curr_connections: 11
        curr_conns_on_port_11211: 3
        curr_items: 100
        curr_items_tot: 100
        curr_temp_items: 0
        ep_concurrentDB: 1
        ep_store_max_concurrency: 10
        vb_active_curr_items: 100
        vb_pending_curr_items: 0
        vb_replica_curr_items: 0

        Show
        jin Jin Lim (Inactive) added a comment - Downloaded the latest build and src code from the main trunk. "net start couchbaseserver" still fails starting memcached. The log indicates that memcached exited right after the warmup. However, I was able to start stand-alone memcached with the same ep.so library and finished running some I/Os. See the compiled output below: c:\Program Files\Couchbase\Server\bin>memcached -p 11211 -E ..\lib\memcached\ep. so -r -e "ht_size=12289;ht_locks=23;min_data_age=1;dbname=..\var\lib\couchbase\d ata\default;queue_age_cap=5;max_size=84901888;vb0=true;backend=couchdb" -u nobody Warning: failed to load the engine session stats due to IO exception "basic_ios ::clear" -> This is a normal error output for no stats.json file being found during the initial startup! Failed to load mutation log, falling back to key dump metadata loaded in 2749 usec warmup completed in 3601 usec Jins-MacBook-Pro:bin jin$ ./memcachetest -h 10.3.121.142:11211 -i 100 -V Average with 1 threads Get operations: #of ops. min max avg max90th max95th max99th 6716 1504 us 4555 ms 22 ms 40 ms 54 ms 110 ms Set operations: #of ops. min max avg max90th max95th max99th 3284 1699 us 937 ms 25 ms 46 ms 60 ms 151 ms Usr: 0.238064 Sys: 1.037286 Tot: 236.509054 Server time: Usr: 0.000000 Sys: 0.000000 Total gets: 0 Total sets: 100 Jins-MacBook-Pro:bin jin$ ./cbstats 10.3.121.142:11211 timings data_age (1682 total) 1s - 2s : ( 74.97%) 1261 ####################################################### 2s - 4s : (100.00%) 421 ################## disk_commit (143909 total) 0 - 1s : (100.00%) 143909 ######################################################################## disk_insert (100 total) 8us - 16us : ( 31.00%) 31 ####################### 16us - 32us : ( 70.00%) 39 ############################# 32us - 64us : ( 98.00%) 28 ##################### 64us - 128us : ( 99.00%) 1 256us - 512us : (100.00%) 1 disk_update (1582 total) 0 - 1us : ( 0.06%) 1 8us - 16us : ( 18.27%) 288 ############# 16us - 32us : ( 58.03%) 629 ############################# 32us - 64us : ( 99.87%) 662 ############################### 64us - 128us : (100.00%) 2 disk_vbstate_snapshot (1 total) 1ms - 2ms : (100.00%) 1 ############################################################################# get_cmd (6716 total) 0 - 1us : ( 0.07%) 5 8us - 16us : ( 33.41%) 2239 ######################## 16us - 32us : ( 97.56%) 4308 ############################################### 32us - 64us : ( 99.82%) 152 # 64us - 128us : ( 99.99%) 11 128us - 256us : (100.00%) 1 get_stats_cmd (13 total) 64us - 128us : ( 7.69%) 1 ##### 128us - 256us : ( 46.15%) 5 ############################# 1ms - 2ms : ( 53.85%) 1 ##### 2ms - 4ms : (100.00%) 6 ################################### item_alloc_sizes (3384 total) 32 - 64 : ( 0.89%) 30 64 - 128 : ( 4.05%) 107 ## 128 - 256 : ( 7.36%) 112 ## 256 - 512 : ( 13.09%) 194 #### 512 - 1KB : ( 28.99%) 538 ########### 1KB - 2KB : ( 50.35%) 723 ############### 2KB - 4KB : (100.00%) 1680 #################################### storage_age (1682 total) 0 - 1s : ( 74.32%) 1250 ###################################################### 1s - 2s : (100.00%) 432 ################### Jins-MacBook-Pro:bin jin$ ./cbstats 10.3.121.142:11211 all | grep curr curr_connections: 11 curr_conns_on_port_11211: 3 curr_items: 100 curr_items_tot: 100 curr_temp_items: 0 ep_concurrentDB: 1 ep_store_max_concurrency: 10 vb_active_curr_items: 100 vb_pending_curr_items: 0 vb_replica_curr_items: 0
        Hide
        jin Jin Lim (Inactive) added a comment -

        It appears to be that it is not an ep_engine issue since we can start memcached with the ep library without any issue and do some I/Os.

        Show
        jin Jin Lim (Inactive) added a comment - It appears to be that it is not an ep_engine issue since we can start memcached with the ep library without any issue and do some I/Os.
        Hide
        sharon Sharon Barr (Inactive) added a comment -

        Hi there.
        it's a blocker bug. if it is not an ep-engine issue, then it should be passed to the next owner/component. no update since last Thursday..

        Show
        sharon Sharon Barr (Inactive) added a comment - Hi there. it's a blocker bug. if it is not an ep-engine issue, then it should be passed to the next owner/component. no update since last Thursday..
        Hide
        peter peter added a comment -

        Bin, thank you for looking into this.

        Show
        peter peter added a comment - Bin, thank you for looking into this.
        peter peter made changes -
        Assignee Jin Lim [ jin ] Bin Cui [ bcui ]
        peter peter made changes -
        Component/s bucket-engine [ 10010 ]
        Component/s couchbase-bucket [ 10173 ]
        steve Steve Yen made changes -
        Sprint Priority 1
        Hide
        bcui Bin Cui added a comment -

        The last workable build was 1486. After that, we build snappy and couchstore as dll and ep.so link them as dlls.
        Test 1: Rollback to static linking. Rebuild snappy and couchstore as static libraries. so ep.so will link them as static ones. However, the problem still exists.
        Test 2: Using windbg, i observed that first chance exceptions triggerred when loading bucket_engine.so and ep.so.

        Show
        bcui Bin Cui added a comment - The last workable build was 1486. After that, we build snappy and couchstore as dll and ep.so link them as dlls. Test 1: Rollback to static linking. Rebuild snappy and couchstore as static libraries. so ep.so will link them as static ones. However, the problem still exists. Test 2: Using windbg, i observed that first chance exceptions triggerred when loading bucket_engine.so and ep.so.
        Hide
        bcui Bin Cui added a comment -

        Test 3: Based on build 1486. Replace memcached.exe and bucket-engine.so with the latest build, it still works. But memcached.exe will crash as soon as ep.so is replaced with the latest one.

        Show
        bcui Bin Cui added a comment - Test 3: Based on build 1486. Replace memcached.exe and bucket-engine.so with the latest build, it still works. But memcached.exe will crash as soon as ep.so is replaced with the latest one.
        Hide
        bcui Bin Cui added a comment -

        The root cause of crashing is narrowed down to commit ccaa609dc02493b5d4dcee842a13dae8439f9714 on ep-engine, i.e. CBD-458 Add support for configurable dispatcher task start time, which was checked in at Jul 24. After reverting the change, i no longer see memcached crash any more. Chiyoung is investigating it now.

        Show
        bcui Bin Cui added a comment - The root cause of crashing is narrowed down to commit ccaa609dc02493b5d4dcee842a13dae8439f9714 on ep-engine, i.e. CBD-458 Add support for configurable dispatcher task start time, which was checked in at Jul 24. After reverting the change, i no longer see memcached crash any more. Chiyoung is investigating it now.
        Hide
        peter peter added a comment -

        Chiyoung is investigating the issue. Thank you.

        Show
        peter peter added a comment - Chiyoung is investigating the issue. Thank you.
        peter peter made changes -
        Assignee Bin Cui [ bcui ] Chiyoung Seo [ chiyoung ]
        peter peter made changes -
        Component/s couchbase-bucket [ 10173 ]
        Component/s bucket-engine [ 10010 ]
        Show
        chiyoung Chiyoung Seo added a comment - http://review.couchbase.org/#change,19454
        chiyoung Chiyoung Seo made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        thuan Thuan Nguyen added a comment -

        Integrated in github-ep-engine-2-0 #392 (See http://qa.hq.northscale.net/job/github-ep-engine-2-0/392/)
        MB-6057 Disable access scanner scheduling at a specific time (Revision 6c0952133127c583adb25a87a3c84d97875d534a)

        Result = SUCCESS
        Chiyoung Seo :
        Files :

        • configuration.json
        Show
        thuan Thuan Nguyen added a comment - Integrated in github-ep-engine-2-0 #392 (See http://qa.hq.northscale.net/job/github-ep-engine-2-0/392/ ) MB-6057 Disable access scanner scheduling at a specific time (Revision 6c0952133127c583adb25a87a3c84d97875d534a) Result = SUCCESS Chiyoung Seo : Files : configuration.json
        mikew Mike Wiederhold made changes -
        Sprint Status Current Sprint
        Hide
        thuan Thuan Nguyen added a comment -

        Integrated in github-ep-engine-2-0 #420 (See http://qa.hq.northscale.net/job/github-ep-engine-2-0/420/)
        Revert "MB-6057 Disable access scanner scheduling at a specific time" (Revision 64b4818f10aa52f9c9bf3f4f6ed5a3b27f1808aa)

        Result = SUCCESS
        Mike Wiederhold :
        Files :

        • configuration.json
        Show
        thuan Thuan Nguyen added a comment - Integrated in github-ep-engine-2-0 #420 (See http://qa.hq.northscale.net/job/github-ep-engine-2-0/420/ ) Revert " MB-6057 Disable access scanner scheduling at a specific time" (Revision 64b4818f10aa52f9c9bf3f4f6ed5a3b27f1808aa) Result = SUCCESS Mike Wiederhold : Files : configuration.json
        farshid Farshid Ghods (Inactive) made changes -
        Labels trunk-green-blockers
        farshid Farshid Ghods (Inactive) made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            chiyoung Chiyoung Seo
            Reporter:
            iryna iryna
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes