Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-6638

beam.smp ( erl_bestfit_alloc.c:435 ) segmentation fault when rebalancing a new node to a cluster with 2 buckets and 1 design doc per bucket

    Details

      Description

      Cluster information:

      • Load 9 million items to both buckets
      • Cluster has 2 buckets, default (3GB) and saslbucket (3GB)
      • Each bucket has one doc and 2 views for each doc (default d1 and saslbucket d11)
      • Create cluster with 10 nodes installed couchbase server 2.0.0-1708

      10.3.121.13
      10.3.121.14
      10.3.121.15
      10.3.121.16
      10.3.121.17
      10.3.121.20
      10.3.121.22
      10.3.121.24
      10.3.121.25
      10.3.121.23

      • Data path /data
      • View path /data

      Doing rebalanc in, out, failover.
      Last failed was rebalance add 2 nodes, 23 and 24 into cluster.
      During rebalance, node 23 suddenly down. Rebalance failed as expected.
      Then later, memcached on node 13 was killed and restart.

      [ns_server:debug,2012-09-12T17:06:05.705,ns_1@10.3.121.13:<0.28110.33>:ns_pubsub:do_subscribe_link:134]Parent process of subscription

      {ns_config_events,<0.28109.33>} exited with reason {timeout,
      {gen_server,
      call,
      [ns_node_disco,
      nodes_wanted]}}
      [ns_server:debug,2012-09-12T17:06:05.712,ns_1@10.3.121.13:<0.28166.33>:ns_pubsub:do_subscribe_link:134]Parent process of subscription {ns_config_events,<0.28165.33>} exited with reason shutdown
      [ns_server:error,2012-09-12T17:06:05.718,ns_1@10.3.121.13:<0.29417.33>:ns_memcached:verify_report_long_call:274]call {stats,<<>>} took too long: 5881903 us
      [ns_server:debug,2012-09-12T17:06:05.720,ns_1@10.3.121.13:<0.28132.33>:ns_pubsub:do_subscribe_link:134]Parent process of subscription {ns_node_disco_events,<0.28130.33>} exited with reason shutdown
      [ns_server:debug,2012-09-12T17:06:05.732,ns_1@10.3.121.13:<0.28110.33>:ns_pubsub:do_subscribe_link:149]Deleting {ns_config_events,<0.28109.33>}

      event handler: ok
      [ns_server:debug,2012-09-12T17:06:05.733,ns_1@10.3.121.13:<0.28166.33>:ns_pubsub:do_subscribe_link:149]Deleting

      {ns_config_events,<0.28165.33>}

      event handler: ok
      [ns_server:debug,2012-09-12T17:06:05.739,ns_1@10.3.121.13:<0.5306.465>:compaction_daemon:do_chain_compactors:500]Got exit signal from parent:

      {'EXIT',<0.4974.465>,shutdown}

      [ns_server:debug,2012-09-12T17:06:05.740,ns_1@10.3.121.13:<0.28132.33>:ns_pubsub:do_subscribe_link:149]Deleting

      {ns_node_disco_events,<0.28130.33>}

      event handler: ok
      [ns_server:debug,2012-09-12T17:06:05.740,ns_1@10.3.121.13:ns_node_disco:ns_node_disco:init:103]Initting ns_node_disco with ['ns_1@10.3.121.15','ns_1@10.3.121.16',
      'ns_1@10.3.121.17','ns_1@10.3.121.22',
      'ns_1@10.3.121.20','ns_1@10.3.121.14',
      'ns_1@10.3.121.25','ns_1@10.3.121.24']
      [ns_server:debug,2012-09-12T17:06:05.741,ns_1@10.3.121.13:<0.28129.33>:ns_pubsub:do_subscribe_link:134]Parent process of subscription

      {ns_config_events,<0.28128.33>} exited with reason shutdown
      [ns_server:debug,2012-09-12T17:06:05.743,ns_1@10.3.121.13:<0.19269.466>:janitor_agent:new_style_query_vbucket_states_loop:111]Exception from query_vbucket_states of "saslbucket":'ns_1@10.3.121.23'
      {'EXIT',{{nodedown,'ns_1@10.3.121.23'},
      {gen_server,call,
      [{'janitor_agent-saslbucket','ns_1@10.3.121.23'},
      query_vbucket_states,infinity]}}}
      [ns_server:debug,2012-09-12T17:06:05.745,ns_1@10.3.121.13:<0.5309.465>:compaction_daemon:do_chain_compactors:500]Got exit signal from parent: {'EXIT',<0.5306.465>,shutdown}
      [ns_server:debug,2012-09-12T17:06:05.746,ns_1@10.3.121.13:<0.28126.33>:ns_pubsub:do_subscribe_link:134]Parent process of subscription {ns_config_events,<0.28125.33>} exited with reason shutdown
      [ns_server:debug,2012-09-12T17:06:05.761,ns_1@10.3.121.13:<0.19269.466>:janitor_agent:new_style_query_vbucket_states_loop_next_step:116]Waiting for "saslbucket" on 'ns_1@10.3.121.23'
      [ns_server:debug,2012-09-12T17:06:05.897,ns_1@10.3.121.13:<0.28129.33>:ns_pubsub:do_subscribe_link:149]Deleting {ns_config_events,<0.28128.33>}

      event handler: ok
      [ns_server:debug,2012-09-12T17:06:05.898,ns_1@10.3.121.13:<0.14047.466>:compaction_daemon:do_spawn_view_index_compactor:742]Got exit signal from parent:

      {'EXIT',<0.5309.465>,shutdown}

      [ns_server:debug,2012-09-12T17:06:05.903,ns_1@10.3.121.13:ns_cookie_manager:ns_cookie_manager:do_cookie_sync:115]ns_cookie_manager do_cookie_sync
      [ns_server:debug,2012-09-12T17:06:05.903,ns_1@10.3.121.13:<0.28126.33>:ns_pubsub:do_subscribe_link:149]Deleting

      {ns_config_events,<0.28125.33>}

      event handler: ok
      [couchdb:info,2012-09-12T17:06:05.906,ns_1@10.3.121.13:<0.5719.436>:couch_log:info:39]Set view `default`, replica group `_design/d1`, canceling compaction (pid <0.18999.466>)
      [ns_server:debug,2012-09-12T17:06:06.000,ns_1@10.3.121.13:ns_cookie_manager:ns_cookie_manager:do_cookie_save:152]saving cookie to "/opt/couchbase/var/lib/couchbase/couchbase-server.cookie"
      [ns_server:debug,2012-09-12T17:06:06.002,ns_1@10.3.121.13:<0.28122.33>:ns_pubsub:do_subscribe_link:134]Parent process of subscription

      {ns_stats_event,<0.28121.33>} exited with reason shutdown
      [ns_server:debug,2012-09-12T17:06:06.009,ns_1@10.3.121.13:<0.28122.33>:ns_pubsub:do_subscribe_link:149]Deleting {ns_stats_event,<0.28121.33>}

      event handler: ok
      [ns_server:debug,2012-09-12T17:06:06.040,ns_1@10.3.121.13:<0.28120.33>:ns_pubsub:do_subscribe_link:134]Parent process of subscription

      {ns_tick_event,<0.28119.33>} exited with reason shutdown
      [ns_server:debug,2012-09-12T17:06:06.044,ns_1@10.3.121.13:<0.28648.34>:single_bucket_sup:top_loop:28]Delegating exit {'EXIT',<0.28117.33>,shutdown} to child supervisor: <0.28649.34>

      [ns_server:debug,2012-09-12T17:06:06.047,ns_1@10.3.121.13:ns_cookie_manager:ns_cookie_manager:do_cookie_save:154]attempted to save cookie to "/opt/couchbase/var/lib/couchbase/couchbase-server.cookie": ok
      [ns_server:debug,2012-09-12T17:06:06.074,ns_1@10.3.121.13:<0.28120.33>:ns_pubsub:do_subscribe_link:149]Deleting {ns_tick_event,<0.28119.33>}

      event handler: ok
      [ns_server:debug,2012-09-12T17:06:06.074,ns_1@10.3.121.13:<0.19276.466>:ns_node_disco:do_nodes_wanted_updated_fun:202]ns_node_disco: nodes_wanted updated: ['ns_1@10.3.121.13','ns_1@10.3.121.14',
      'ns_1@10.3.121.15','ns_1@10.3.121.16',
      'ns_1@10.3.121.17','ns_1@10.3.121.20',
      'ns_1@10.3.121.22','ns_1@10.3.121.23',
      'ns_1@10.3.121.24','ns_1@10.3.121.25'], with cookie: oetfisqvqrrxvaat
      [ns_server:debug,2012-09-12T17:06:06.079,ns_1@10.3.121.13:<0.28689.34>:ns_pubsub:do_subscribe_link:134]Parent process of subscription

      {ns_stats_event,<0.28688.34>} exited with reason shutdown
      [ns_server:debug,2012-09-12T17:06:06.080,ns_1@10.3.121.13:<0.28687.34>:ns_pubsub:do_subscribe_link:134]Parent process of subscription {ns_tick_event,<0.28686.34>} exited with reason shutdown
      [ns_server:debug,2012-09-12T17:06:06.083,ns_1@10.3.121.13:<0.28689.34>:ns_pubsub:do_subscribe_link:149]Deleting {ns_stats_event,<0.28688.34>}

      event handler: ok
      [ns_server:debug,2012-09-12T17:06:06.084,ns_1@10.3.121.13:<0.28687.34>:ns_pubsub:do_subscribe_link:149]Deleting

      {ns_tick_event,<0.28686.34>}

      event handler: ok
      [stats:warn,2012-09-12T17:06:06.087,ns_1@10.3.121.13:<0.29425.33>:stats_collector:latest_tick:201]Dropped 7 ticks
      [rebalance:info,2012-09-12T17:06:06.091,ns_1@10.3.121.13:<0.25334.457>:ebucketmigrator_srv:do_confirm_sent_messages:655]Got close ack!

      [rebalance:info,2012-09-12T17:06:06.120,ns_1@10.3.121.13:<0.23919.457>:ebucketmigrator_srv:do_confirm_sent_messages:655]Got close ack!

      [ns_server:debug,2012-09-12T17:06:06.120,ns_1@10.3.121.13:<0.1936.453>:ns_pubsub:do_subscribe_link:149]Deleting

      {master_activity_events_ingress,<0.28014.33>}

      event handler:

      {error, module_not_found}

      [rebalance:info,2012-09-12T17:06:06.128,ns_1@10.3.121.13:<0.23192.457>:ebucketmigrator_srv:do_confirm_sent_messages:655]Got close ack!

      [rebalance:info,2012-09-12T17:06:06.147,ns_1@10.3.121.13:<0.22947.457>:ebucketmigrator_srv:do_confirm_sent_messages:655]Got close ack!

      [rebalance:info,2012-09-12T17:06:06.201,ns_1@10.3.121.13:<0.21996.457>:ebucketmigrator_srv:do_confirm_sent_messages:655]Got close ack!

      [rebalance:info,2012-09-12T17:06:06.218,ns_1@10.3.121.13:<0.11912.457>:ebucketmigrator_srv:do_confirm_sent_messages:655]Got close ack!

      [rebalance:info,2012-09-12T17:06:06.223,ns_1@10.3.121.13:<0.10815.457>:ebucketmigrator_srv:do_confirm_sent_messages:655]Got close ack!

      [ns_server:debug,2012-09-12T17:06:06.228,ns_1@10.3.121.13:<0.28050.33>:ns_pubsub:do_subscribe_link:134]Parent process of subscription

      {buckets_events,<0.28049.33>} exited with reason {shutdown,
      {gen_server,
      call,
      ['ns_vbm_new_sup-default',
      which_children,
      infinity]}}
      [ns_server:debug,2012-09-12T17:06:06.239,ns_1@10.3.121.13:<0.28050.33>:ns_pubsub:do_subscribe_link:149]Deleting {buckets_events,<0.28049.33>}

      event handler: ok
      [user:info,2012-09-12T17:06:06.245,ns_1@10.3.121.13:'ns_memcached-default':ns_memcached:terminate:625]Shutting down bucket "default" on 'ns_1@10.3.121.13' for server shutdown
      [ns_server:info,2012-09-12T17:06:06.261,ns_1@10.3.121.13:'ns_memcached-default':ns_memcached:terminate:636]This bucket shutdown is not due to bucket deletion. Doing nothing
      [ns_server:debug,2012-09-12T17:06:06.276,ns_1@10.3.121.13:<0.28651.34>:ns_pubsub:do_subscribe_link:134]Parent process of subscription

      {ns_config_events,<0.28650.34>} exited with reason shutdown
      [ns_server:debug,2012-09-12T17:06:06.276,ns_1@10.3.121.13:<0.28663.34>:ns_pubsub:do_subscribe_link:134]Parent process of subscription {mc_couch_events,<0.28650.34>} exited with reason shutdown
      [ns_server:debug,2012-09-12T17:06:06.276,ns_1@10.3.121.13:<0.28662.34>:ns_pubsub:do_subscribe_link:134]Parent process of subscription {ns_node_disco_events,<0.28650.34>} exited with reason shutdown
      [ns_server:debug,2012-09-12T17:06:06.276,ns_1@10.3.121.13:<0.28648.34>:single_bucket_sup:top_loop:24]per-bucket supervisor for "default" died with reason shutdown

      [ns_server:debug,2012-09-12T17:06:06.279,ns_1@10.3.121.13:<0.28651.34>:ns_pubsub:do_subscribe_link:149]Deleting {ns_config_events,<0.28650.34>}

      event handler: ok
      [ns_server:debug,2012-09-12T17:06:06.281,ns_1@10.3.121.13:<0.28663.34>:ns_pubsub:do_subscribe_link:149]Deleting

      {mc_couch_events,<0.28650.34>}

      event handler: ok
      [ns_server:debug,2012-09-12T17:06:06.283,ns_1@10.3.121.13:<0.28662.34>:ns_pubsub:do_subscribe_link:149]Deleting

      {ns_node_disco_events,<0.28650.34>}

      event handler: ok
      [ns_server:debug,2012-09-12T17:06:06.283,ns_1@10.3.121.13:<0.29393.33>:single_bucket_sup:top_loop:28]Delegating exit

      {'EXIT',<0.28117.33>,shutdown}

      to child supervisor: <0.29394.33>

      [ns_server:debug,2012-09-12T17:06:06.299,ns_1@10.3.121.13:<0.29428.33>:ns_pubsub:do_subscribe_link:134]Parent process of subscription

      {ns_stats_event,<0.29427.33>} exited with reason shutdown
      [ns_server:debug,2012-09-12T17:06:06.300,ns_1@10.3.121.13:<0.29426.33>:ns_pubsub:do_subscribe_link:134]Parent process of subscription {ns_tick_event,<0.29425.33>} exited with reason shutdown
      [ns_server:debug,2012-09-12T17:06:06.305,ns_1@10.3.121.13:<0.29428.33>:ns_pubsub:do_subscribe_link:149]Deleting {ns_stats_event,<0.29427.33>}

      event handler: ok
      [ns_server:debug,2012-09-12T17:06:06.307,ns_1@10.3.121.13:<0.29426.33>:ns_pubsub:do_subscribe_link:149]Deleting

      {ns_tick_event,<0.29425.33>}

      event handler: ok
      [rebalance:info,2012-09-12T17:06:06.310,ns_1@10.3.121.13:<0.11978.466>:ebucketmigrator_srv:do_confirm_sent_messages:655]Got close ack!

      [ns_server:info,2012-09-12T17:06:06.323,ns_1@10.3.121.13:ns_port_memcached:ns_port_server:log:169]memcached<0.28094.33>: Thu Sep 13 00:06:06.117323 3: TAP (Consumer) eq_tapq:anon_509 - disconnected
      memcached<0.28094.33>: Thu Sep 13 00:06:06.123981 3: TAP (Consumer) eq_tapq:anon_1652 - disconnected
      memcached<0.28094.33>: Thu Sep 13 00:06:06.144392 3: TAP (Consumer) eq_tapq:anon_480 - disconnected
      memcached<0.28094.33>: Thu Sep 13 00:06:06.151955 3: TAP (Consumer) eq_tapq:anon_488 - disconnected
      memcached<0.28094.33>: Thu Sep 13 00:06:06.203038 3: TAP (Consumer) eq_tapq:anon_504 - disconnected
      memcached<0.28094.33>: Thu Sep 13 00:06:06.221683 3: TAP (Consumer) eq_tapq:anon_719 - disconnected
      memcached<0.28094.33>: Thu Sep 13 00:06:06.227307 3: TAP (Consumer) eq_tapq:anon_499 - disconnected

      memcached log from node 13 at the time it was killed

      Thu Sep 13 00:06:07.606021 3: Schedule cleanup of "eq_tapq:anon_20"
      Thu Sep 13 00:06:07.606058 3: Schedule cleanup of "eq_tapq:anon_34"
      Thu Sep 13 00:06:07.606083 3: Schedule cleanup of "eq_tapq:anon_36"
      Thu Sep 13 00:06:07.606115 3: Schedule cleanup of "eq_tapq:anon_40"
      Thu Sep 13 00:06:07.606142 3: Schedule cleanup of "eq_tapq:anon_730"
      Thu Sep 13 00:06:07.606165 3: Schedule cleanup of "eq_tapq:anon_2425"
      Thu Sep 13 00:06:21.096648 3: Had to wait 13 s for shutdown
      Thu Sep 13 00:06:31.465158 3: Shutting down tap connections!
      Thu Sep 13 00:06:31.465199 3: Schedule cleanup of "eq_tapq:replication_ns_1@10.3.121.15"
      Thu Sep 13 00:06:31.465278 3: Schedule cleanup of "eq_tapq:replication_ns_1@10.3.121.16"
      Thu Sep 13 00:06:31.465357 3: Schedule cleanup of "eq_tapq:replication_ns_1@10.3.121.22"
      Thu Sep 13 00:06:31.465405 3: Schedule cleanup of "eq_tapq:replication_ns_1@10.3.121.17"
      Thu Sep 13 00:06:31.465414 3: TAP (Producer) eq_tapq:replication_ns_1@10.3.121.15 - Clear the tap queues by force
      Thu Sep 13 00:06:31.465441 3: Schedule cleanup of "eq_tapq:replication_ns_1@10.3.121.20"
      Thu Sep 13 00:06:31.465471 3: Schedule cleanup of "eq_tapq:replication_ns_1@10.3.121.14"
      Thu Sep 13 00:06:31.465492 3: Schedule cleanup of "eq_tapq:replication_ns_1@10.3.121.25"
      Thu Sep 13 00:06:31.465517 3: Schedule cleanup of "eq_tapq:replication_ns_1@10.3.121.24"
      Thu Sep 13 00:06:31.465604 3: TAP (Producer) eq_tapq:replication_ns_1@10.3.121.16 - Clear the tap queues by force
      Thu Sep 13 00:06:31.467071 3: TAP (Producer) eq_tapq:replication_ns_1@10.3.121.22 - Clear the tap queues by force
      Thu Sep 13 00:06:31.467363 3: TAP (Producer) eq_tapq:replication_ns_1@10.3.121.20 - Clear the tap queues by force
      Thu Sep 13 00:06:31.467606 3: TAP (Producer) eq_tapq:replication_ns_1@10.3.121.24 - Clear the tap queues by force
      Thu Sep 13 00:06:31.467710 3: TAP (Producer) eq_tapq:replication_ns_1@10.3.121.25 - Clear the tap queues by force
      Thu Sep 13 00:06:31.467917 3: TAP (Producer) eq_tapq:replication_ns_1@10.3.121.17 - Clear the tap queues by force
      Thu Sep 13 00:06:31.468102 3: TAP (Producer) eq_tapq:replication_ns_1@10.3.121.14 - Clear the tap queues by force

      Link to diags of all nodes https://s3.amazonaws.com/packages.couchbase/diag-logs/orange/201209/10nodes-1708-memcached-killed-after-reb-failed-20120912.tgz

      Link to atop of node 13 https://s3.amazonaws.com/packages.couchbase/atop-files/orange/201209/atop-node13-20120912

      1. memcached.log.6.txt.gz
        217 kB
        Thuan Nguyen
      2. memcached.log.7.txt.gz
        27 kB
        Thuan Nguyen
      # Subject Project Status CR V
      For Gerrit Dashboard: &For+MB-6638=message:MB-6638

        Activity

        thuan Thuan Nguyen created issue -
        chiyoung Chiyoung Seo made changes -
        Field Original Value New Value
        Sprint Status Current Sprint
        Hide
        chiyoung Chiyoung Seo added a comment -

        Node 10.3.121.23, which was newly added, became unreachable from 16:57:37 to 17:38:36. I didn't see any logs in the diags on this node during this period:

        [views:info,2012-09-12T16:57:37.570,ns_1@10.3.121.23:'capi_set_view_manager-saslbucket':capi_set_view_manager:apply_index_states:464]
        Calling couch_set_view:add_replica_partitions([<<"saslbucket">>,
        <<"_design/d11">>,
        [18,19,32,33,34,35,44,45,46,
        105,106,107,114,120,152,153,
        158,159,160,181,182,198,199,
        216,217,224,225,226,227,228,
        229,230,242,245,253,276,277,
        278,279,280,281,282,283,300,
        301,302,303,324,356,357,358,
        359,360,361,381,382,383,384,
        385,443,444,445,451,452,453,
        454,455,463,464,465,466,467,
        468,481,482,483,506,507,508,
        520,521,522,523,524,527,552,
        553,554,565,574,575,576,577,
        578,579,591,592,594,595,596,
        597,602,603,604,619,620,621,
        627,628,629,632,633,634,635,
        636,637,656,657,729,730,731,
        732,749,750,761,808,809,810,
        811,812,836,842,848,849,850,
        851,873,874,875,907]])
        [ns_server:info,2012-09-12T17:38:36.992,nonode@nohost:<0.60.0>:ns_server:init_logging:198]Started & configured logging
        [ns_server:info,2012-09-12T17:38:37.003,nonode@nohost:<0.60.0>:ns_server:log_pending:30]Static config terms:

        Show
        chiyoung Chiyoung Seo added a comment - Node 10.3.121.23, which was newly added, became unreachable from 16:57:37 to 17:38:36. I didn't see any logs in the diags on this node during this period: [views:info,2012-09-12T16:57:37.570,ns_1@10.3.121.23:'capi_set_view_manager-saslbucket':capi_set_view_manager:apply_index_states:464] Calling couch_set_view:add_replica_partitions([<<"saslbucket">>, <<"_design/d11">>, [18,19,32,33,34,35,44,45,46, 105,106,107,114,120,152,153, 158,159,160,181,182,198,199, 216,217,224,225,226,227,228, 229,230,242,245,253,276,277, 278,279,280,281,282,283,300, 301,302,303,324,356,357,358, 359,360,361,381,382,383,384, 385,443,444,445,451,452,453, 454,455,463,464,465,466,467, 468,481,482,483,506,507,508, 520,521,522,523,524,527,552, 553,554,565,574,575,576,577, 578,579,591,592,594,595,596, 597,602,603,604,619,620,621, 627,628,629,632,633,634,635, 636,637,656,657,729,730,731, 732,749,750,761,808,809,810, 811,812,836,842,848,849,850, 851,873,874,875,907]]) [ns_server:info,2012-09-12T17:38:36.992,nonode@nohost:<0.60.0>:ns_server:init_logging:198] Started & configured logging [ns_server:info,2012-09-12T17:38:37.003,nonode@nohost:<0.60.0>:ns_server:log_pending:30] Static config terms:
        chiyoung Chiyoung Seo made changes -
        Summary [longevity] memcached killed after rebalance failed [longevity] One of nodes, which was newly added, suddenly went down during rebalance while updating and querying views
        Hide
        chiyoung Chiyoung Seo added a comment -

        Alk, please see my comment and advise me why this can happen.

        Show
        chiyoung Chiyoung Seo added a comment - Alk, please see my comment and advise me why this can happen.
        chiyoung Chiyoung Seo made changes -
        Assignee Chiyoung Seo [ chiyoung ] Aleksey Kondratenko [ alkondratenko ]
        chiyoung Chiyoung Seo made changes -
        Component/s ns_server [ 10019 ]
        Component/s view-engine [ 10060 ]
        Hide
        karan Karan Kumar (Inactive) added a comment -

        I was able to figure out some more stuff.
        The beam.smp segfaulted around this time. Which caused all sorts of query/rebalance failures.

        Sep 12 16:57:37 orange-3006 kernel: beam.smp[25844]: segfault at 10000000f ip 000000000043ef50 sp 00007f7139064d48 error 6 in beam.smp[400000+1d6000]

        Show
        karan Karan Kumar (Inactive) added a comment - I was able to figure out some more stuff. The beam.smp segfaulted around this time. Which caused all sorts of query/rebalance failures. Sep 12 16:57:37 orange-3006 kernel: beam.smp [25844] : segfault at 10000000f ip 000000000043ef50 sp 00007f7139064d48 error 6 in beam.smp [400000+1d6000]
        Hide
        karan Karan Kumar (Inactive) added a comment -

        Unfortunately, I dont see any core file on the system.

        Even though, cores are enabled on the machine.

        Show
        karan Karan Kumar (Inactive) added a comment - Unfortunately, I dont see any core file on the system. Even though, cores are enabled on the machine.
        karan Karan Kumar (Inactive) made changes -
        Summary [longevity] One of nodes, which was newly added, suddenly went down during rebalance while updating and querying views [longevity] Newly added node, went down due to beam.smp segfaulted
        chiyoung Chiyoung Seo made changes -
        Component/s couchbase-bucket [ 10173 ]
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        We'll need core dump here.

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - We'll need core dump here.
        alkondratenko Aleksey Kondratenko (Inactive) made changes -
        Assignee Aleksey Kondratenko [ alkondratenko ] Thuan Nguyen [ thuan ]
        Hide
        karan Karan Kumar (Inactive) added a comment -

        We are hitting this again.

        But for some reason the cores are not getting generated.

        beam.smp[16816]: segfault at 0 ip 00000030af08847b sp 00007f011e38fdd8 error 4 in libc-2.12.so[30af000000+186000]

        Show
        karan Karan Kumar (Inactive) added a comment - We are hitting this again. But for some reason the cores are not getting generated. beam.smp [16816] : segfault at 0 ip 00000030af08847b sp 00007f011e38fdd8 error 4 in libc-2.12.so [30af000000+186000]
        Hide
        karan Karan Kumar (Inactive) added a comment -

        Increasing this to blocker
        Hitting this more often than not

        Show
        karan Karan Kumar (Inactive) added a comment - Increasing this to blocker Hitting this more often than not
        karan Karan Kumar (Inactive) made changes -
        Priority Major [ 3 ] Blocker [ 1 ]
        Hide
        karan Karan Kumar (Inactive) added a comment -

        ulimit for all users have now been set to unlimited. and kernel.core_pattern = /data/core.%e.%p on all the nodes.

        Show
        karan Karan Kumar (Inactive) added a comment - ulimit for all users have now been set to unlimited. and kernel.core_pattern = /data/core.%e.%p on all the nodes.
        Hide
        karan Karan Kumar (Inactive) added a comment -

        GNU gdb (GDB) Red Hat Enterprise Linux (7.2-50.el6)
        Copyright (C) 2010 Free Software Foundation, Inc.
        License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
        This is free software: you are free to change and redistribute it.
        There is NO WARRANTY, to the extent permitted by law. Type "show copying"
        and "show warranty" for details.
        This GDB was configured as "x86_64-redhat-linux-gnu".
        For bug reporting instructions, please see:
        <http://www.gnu.org/software/gdb/bugs/>...
        Reading symbols from /opt/couchbase/lib/erlang/erts-5.8.5/bin/beam.smp...done.
        [New Thread 28408]
        [New Thread 28409]
        [New Thread 28410]
        [New Thread 28412]
        [New Thread 28414]
        [New Thread 28420]
        [New Thread 28979]
        [New Thread 28398]
        [New Thread 28400]
        [New Thread 28425]
        [New Thread 28413]
        [New Thread 28416]
        [New Thread 28417]
        [New Thread 28433]
        [New Thread 28589]
        [New Thread 29574]
        [New Thread 28401]
        [New Thread 28405]
        [New Thread 28406]
        [New Thread 28424]
        [New Thread 28415]
        [New Thread 28419]
        [New Thread 29407]
        [New Thread 29576]
        [New Thread 28411]
        [New Thread 28418]
        [New Thread 28421]
        [New Thread 28403]
        [New Thread 28407]
        [New Thread 28423]
        [New Thread 28422]
        Reading symbols from /lib64/libutil.so.1...(no debugging symbols found)...done.
        Loaded symbols for /lib64/libutil.so.1
        Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
        Loaded symbols for /lib64/libdl.so.2
        Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
        Loaded symbols for /lib64/libm.so.6
        Reading symbols from /lib64/libncurses.so.5...(no debugging symbols found)...done.
        Loaded symbols for /lib64/libncurses.so.5
        Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
        [Thread debugging using libthread_db enabled]
        Loaded symbols for /lib64/libpthread.so.0
        Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
        Loaded symbols for /lib64/librt.so.1
        Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
        Loaded symbols for /lib64/libc.so.6
        Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
        Loaded symbols for /lib64/ld-linux-x86-64.so.2
        Reading symbols from /lib64/libtinfo.so.5...(no debugging symbols found)...done.
        Loaded symbols for /lib64/libtinfo.so.5
        Reading symbols from /opt/couchbase/lib/erlang/lib/crypto-2.0.4/priv/lib/crypto.so...done.
        Loaded symbols for /opt/couchbase/lib/erlang/lib/crypto-2.0.4/priv/lib/crypto.so
        Reading symbols from /usr/lib64/libcrypto.so.6...(no debugging symbols found)...done.
        Loaded symbols for /usr/lib64/libcrypto.so.6
        Reading symbols from /usr/lib64/libssl.so.6...(no debugging symbols found)...done.
        Loaded symbols for /usr/lib64/libssl.so.6
        Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
        Loaded symbols for /lib64/libz.so.1
        Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging symbols found)...done.
        Loaded symbols for /lib64/libgssapi_krb5.so.2
        Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols found)...done.
        Loaded symbols for /lib64/libkrb5.so.3
        Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...done.
        Loaded symbols for /lib64/libcom_err.so.2
        Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols found)...done.
        Loaded symbols for /lib64/libk5crypto.so.3
        Reading symbols from /lib64/libkrb5support.so.0...(no debugging symbols found)...done.
        Loaded symbols for /lib64/libkrb5support.so.0
        Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done.
        Loaded symbols for /lib64/libkeyutils.so.1
        Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done.
        Loaded symbols for /lib64/libresolv.so.2
        Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done.
        Loaded symbols for /lib64/libselinux.so.1
        Reading symbols from /opt/couchbase/lib/libv8.so...done.
        Loaded symbols for /opt/couchbase/lib/libv8.so
        Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found)...done.
        Loaded symbols for /lib64/libcrypt.so.1
        Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done.
        Loaded symbols for /usr/lib64/libstdc++.so.6
        Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
        Loaded symbols for /lib64/libgcc_s.so.1
        Reading symbols from /lib64/libfreebl3.so...(no debugging symbols found)...done.
        Loaded symbols for /lib64/libfreebl3.so
        Reading symbols from /opt/couchbase/lib/couchdb/erlang/lib/ejson-0.1.0/priv/ejson.so...done.
        Loaded symbols for /opt/couchbase/lib/couchdb/erlang/lib/ejson-0.1.0/priv/ejson.so
        Reading symbols from /opt/couchbase/lib/couchdb/erlang/lib/snappy-1.0.4/priv/snappy_nif.so...done.
        Loaded symbols for /opt/couchbase/lib/couchdb/erlang/lib/snappy-1.0.4/priv/snappy_nif.so
        Reading symbols from /opt/couchbase/lib/couchdb/erlang/lib/mapreduce-1.0/priv/mapreduce_nif.so...done.
        Loaded symbols for /opt/couchbase/lib/couchdb/erlang/lib/mapreduce-1.0/priv/mapreduce_nif.so
        Reading symbols from /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-5282953-git/priv/lib/couch_icu_driver.so...done.
        Loaded symbols for /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-5282953-git/priv/lib/couch_icu_driver.so
        Reading symbols from /opt/couchbase/lib/libicuuc.so.44...done.
        Loaded symbols for /opt/couchbase/lib/libicuuc.so.44
        Reading symbols from /opt/couchbase/lib/libicudata.so.44...(no debugging symbols found)...done.
        Loaded symbols for /opt/couchbase/lib/libicudata.so.44
        Reading symbols from /opt/couchbase/lib/libicui18n.so.44...done.
        Loaded symbols for /opt/couchbase/lib/libicui18n.so.44
        Reading symbols from /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-5282953-git/priv/lib/couch_ejson_compare.so...done.
        Loaded symbols for /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-5282953-git/priv/lib/couch_ejson_compare.so
        Reading symbols from /opt/couchbase/lib/couchdb/erlang/lib/couch_view_parser-1.0/priv/couch_view_parser_nif.so...done.
        Loaded symbols for /opt/couchbase/lib/couchdb/erlang/lib/couch_view_parser-1.0/priv/couch_view_parser_nif.so
        Core was generated by `/opt/couchbase/lib/erlang/erts-5.8.5/bin/beam.smp -A 16 -sbt u -P 327680 -K tru'.
        Program terminated with signal 11, Segmentation fault.
        #0 0x000000000043ec6b in tree_delete (allctr=<value optimized out>, del=0x12c6a80, flags=<value optimized out>) at beam/erl_bestfit_alloc.c:435
        435 beam/erl_bestfit_alloc.c: No such file or directory.
        in beam/erl_bestfit_alloc.c
        Missing separate debuginfos, use: debuginfo-install couchbase-server-2.0.0-1717.x86_64
        (gdb) t a a bt

        Thread 31 (Thread 0x7f4afea01700 (LWP 28422)):
        #0 0x000000397260804d in pthread_join () from /lib64/libpthread.so.0
        #1 0x00007f4af6371887 in v8::internal::ContextSwitcher::StopPreemption () at src/v8threads.cc:445
        #2 0x00007f4af424b7a3 in destroyContext (ctx=0x7f4aac7b4f18) at mapreduce.cc:354
        #3 0x00000000005138fb in nif_resource_dtor (bin=<value optimized out>) at beam/erl_nif.c:1217
        #4 0x000000000048904b in erts_bin_free (offheap=<value optimized out>) at beam/erl_binary.h:303
        #5 erts_cleanup_offheap (offheap=<value optimized out>) at beam/erl_message.c:174
        #6 0x0000000000498a81 in delete_process (p=0x7f4aa286d568, pix_lock=<value optimized out>) at beam/erl_process.c:6621
        #7 continue_exit_process (p=0x7f4aa286d568, pix_lock=<value optimized out>) at beam/erl_process.c:7663
        #8 0x000000000051db94 in terminate_proc (c_p=0x7f4aa286d568, pc=<value optimized out>, reg=<value optimized out>, bf=<value optimized out>) at beam/beam_emu.c:5633
        #9 handle_error (c_p=0x7f4aa286d568, pc=<value optimized out>, reg=<value optimized out>, bf=<value optimized out>) at beam/beam_emu.c:5508
        #10 0x0000000000520cdf in process_main () at beam/beam_emu.c:3273
        #11 0x000000000048d496 in sched_thread_func (vesdp=0x7f4b000eafc0) at beam/erl_process.c:3782
        #12 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff316200a0) at pthread/ethread.c:106
        #13 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
        #14 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

        Thread 30 (Thread 0x7f4afe000700 (LWP 28423)):
        #0 0x0000000000522f94 in process_main () at x86_64-unknown-linux-gnu/opt/smp/beam_hot.h:919
        #1 0x000000000048d496 in sched_thread_func (vesdp=0x7f4b000f9080) at beam/erl_process.c:3782
        #2 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff316200a0) at pthread/ethread.c:106
        #3 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
        #4 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

        Thread 29 (Thread 0x7f4afec23700 (LWP 28407)):
        #0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
        #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
        #2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d8138) at beam/erl_threads.h:697
        #3 async_get (arg=0x7f4aff3d8138) at beam/erl_async.c:207
        #4 async_main (arg=0x7f4aff3d8138) at beam/erl_async.c:274
        #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
        #6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
        #7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

        Thread 28 (Thread 0x7f4affe69700 (LWP 28403)):
        #0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
        #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
        #2 0x0000000000462ad9 in erts_cnd_wait (unused=<value optimized out>) at beam/erl_threads.h:697
        #3 erts_smp_cnd_wait (unused=<value optimized out>) at beam/erl_smp.h:472
        #4 sys_msg_dispatcher_func (unused=<value optimized out>) at beam/erl_trace.c:3238
        #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff316200d0) at pthread/ethread.c:106
        #6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
        #7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

        Thread 27 (Thread 0x7f4afea47700 (LWP 28421)):
        #0 0x000000397260f03d in waitpid () from /lib64/libpthread.so.0
        #1 0x000000000053906f in child_waiter (unused=<value optimized out>) at sys/unix/sys.c:2821
        #2 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620080) at pthread/ethread.c:106
        #3 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
        #4 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

        Thread 26 (Thread 0x7f4afeaad700 (LWP 28418)):
        #0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
        #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
        #2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d8660) at beam/erl_threads.h:697
        #3 async_get (arg=0x7f4aff3d8660) at beam/erl_async.c:207
        #4 async_main (arg=0x7f4aff3d8660) at beam/erl_async.c:274
        #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
        #6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
        #7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

        Thread 25 (Thread 0x7f4afeb9b700 (LWP 28411)):
        --Type <return> to continue, or q <return> to quit--
        #0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
        #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
        #2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d8318) at beam/erl_threads.h:697
        #3 async_get (arg=0x7f4aff3d8318) at beam/erl_async.c:207
        #4 async_main (arg=0x7f4aff3d8318) at beam/erl_async.c:274
        #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
        #6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
        #7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

        Thread 24 (Thread 0x7f4aaed44700 (LWP 29576)):
        #0 0x0000003971eab15d in nanosleep () from /lib64/libc.so.6
        #1 0x0000003971edf124 in usleep () from /lib64/libc.so.6
        #2 0x00007f4af6371848 in v8::internal::ContextSwitcher::Run (this=0x7f4af814d8d0) at src/v8threads.cc:457
        #3 0x00007f4af62f1580 in v8::internal::ThreadEntry (arg=0x7f4af814d8d0) at src/platform-linux.cc:745
        #4 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
        #5 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

        Thread 23 (Thread 0x7f4a94844700 (LWP 29407)):
        #0 0x0000003971eab15d in nanosleep () from /lib64/libc.so.6
        #1 0x0000003971edf124 in usleep () from /lib64/libc.so.6
        #2 0x00007f4af6371848 in v8::internal::ContextSwitcher::Run (this=0x7f4ab41b09d0) at src/v8threads.cc:457
        #3 0x00007f4af62f1580 in v8::internal::ThreadEntry (arg=0x7f4ab41b09d0) at src/platform-linux.cc:745
        #4 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
        #5 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

        Thread 22 (Thread 0x7f4afea8b700 (LWP 28419)):
        #0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
        #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
        #2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d86d8) at beam/erl_threads.h:697
        #3 async_get (arg=0x7f4aff3d86d8) at beam/erl_async.c:207
        #4 async_main (arg=0x7f4aff3d86d8) at beam/erl_async.c:274
        #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
        #6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
        #7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

        Thread 21 (Thread 0x7f4afeb13700 (LWP 28415)):
        #0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
        #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
        #2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d84f8) at beam/erl_threads.h:697
        #3 async_get (arg=0x7f4aff3d84f8) at beam/erl_async.c:207
        #4 async_main (arg=0x7f4aff3d84f8) at beam/erl_async.c:274
        #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
        #6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
        #7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

        Thread 20 (Thread 0x7f4afd5ff700 (LWP 28424)):
        #0 do_erts_alcu_free (type=<value optimized out>, unused=<value optimized out>, p=0x7f4aaf0b8608) at beam/erl_alloc_util.c:2894
        #1 erts_alcu_free_thr_pref (type=<value optimized out>, unused=<value optimized out>, p=0x7f4aaf0b8608) at beam/erl_alloc_util.c:2949
        #2 0x0000000000489022 in erts_free (offheap=<value optimized out>) at beam/erl_alloc.h:226
        #3 erts_bin_free (offheap=<value optimized out>) at beam/erl_binary.h:307
        #4 erts_cleanup_offheap (offheap=<value optimized out>) at beam/erl_message.c:174
        #5 0x0000000000498a81 in delete_process (p=0x7f4aac311310, pix_lock=<value optimized out>) at beam/erl_process.c:6621
        #6 continue_exit_process (p=0x7f4aac311310, pix_lock=<value optimized out>) at beam/erl_process.c:7663
        #7 0x000000000051db94 in terminate_proc (c_p=0x7f4aac311310, pc=<value optimized out>, reg=<value optimized out>, bf=<value optimized out>) at beam/beam_emu.c:5633
        #8 handle_error (c_p=0x7f4aac311310, pc=<value optimized out>, reg=<value optimized out>, bf=<value optimized out>) at beam/beam_emu.c:5508
        #9 0x0000000000520cdf in process_main () at beam/beam_emu.c:3273
        #10 0x000000000048d496 in sched_thread_func (vesdp=0x7f4b00107140) at beam/erl_process.c:3782
        #11 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff316200a0) at pthread/ethread.c:106
        #12 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
        #13 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

        Thread 19 (Thread 0x7f4afec45700 (LWP 28406)):
        #0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
        --Type <return> to continue, or q <return> to quit--
        #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
        #2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d80c0) at beam/erl_threads.h:697
        #3 async_get (arg=0x7f4aff3d80c0) at beam/erl_async.c:207
        #4 async_main (arg=0x7f4aff3d80c0) at beam/erl_async.c:274
        #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
        #6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
        #7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

        Thread 18 (Thread 0x7f4afec67700 (LWP 28405)):
        #0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
        #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
        #2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d8048) at beam/erl_threads.h:697
        #3 async_get (arg=0x7f4aff3d8048) at beam/erl_async.c:207
        #4 async_main (arg=0x7f4aff3d8048) at beam/erl_async.c:274
        #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
        #6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
        #7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

        Thread 17 (Thread 0x7f4b0017c700 (LWP 28401)):
        #0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
        #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
        #2 0x000000000046a1cf in erts_cnd_wait (unused=<value optimized out>) at beam/erl_threads.h:697
        #3 erts_smp_cnd_wait (unused=<value optimized out>) at beam/erl_smp.h:472
        #4 emergency_watchdog (unused=<value optimized out>) at beam/utils.c:3945
        #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff316200d0) at pthread/ethread.c:106
        #6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
        #7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

        Thread 16 (Thread 0x7f4a93e43700 (LWP 29574)):
        #0 0x0000003971eab15d in nanosleep () from /lib64/libc.so.6
        #1 0x0000003971edf124 in usleep () from /lib64/libc.so.6
        #2 0x00007f4af6371848 in v8::internal::ContextSwitcher::Run (this=0x7f4aa8091b00) at src/v8threads.cc:457
        #3 0x00007f4af62f1580 in v8::internal::ThreadEntry (arg=0x7f4aa8091b00) at src/platform-linux.cc:745
        #4 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
        #5 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

        Thread 15 (Thread 0x7f4aaed55700 (LWP 28589)):
        #0 0x000000397260d6c0 in sem_wait () from /lib64/libpthread.so.0
        #1 0x00007f4af62f118d in v8::internal::LinuxSemaphore::Wait (this=0x7f4aa8009b10) at src/platform-linux.cc:869
        #2 0x00007f4af630845c in v8::internal::RuntimeProfiler::WaitForSomeIsolateToEnterJS () at src/runtime-profiler.cc:386
        #3 0x00007f4af62f2136 in v8::internal::SignalSender::Run (this=0x7f4aa8009470) at src/platform-linux.cc:1126
        #4 0x00007f4af62f1580 in v8::internal::ThreadEntry (arg=0x7f4aa8009470) at src/platform-linux.cc:745
        #5 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
        #6 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

        Thread 14 (Thread 0x7f4aaffff700 (LWP 28433)):
        #0 0x0000003971eab15d in nanosleep () from /lib64/libc.so.6
        #1 0x0000003971edf124 in usleep () from /lib64/libc.so.6
        #2 0x00007f4af4249397 in terminatorLoop (args=<value optimized out>) at mapreduce_nif.cc:445
        #3 0x0000000000584d89 in thr_wrapper (vtwd=0x7f4afea00ae0) at pthread/ethread.c:106
        #4 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
        #5 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

        Thread 13 (Thread 0x7f4afeacf700 (LWP 28417)):
        #0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
        #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
        #2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d85e8) at beam/erl_threads.h:697
        #3 async_get (arg=0x7f4aff3d85e8) at beam/erl_async.c:207
        #4 async_main (arg=0x7f4aff3d85e8) at beam/erl_async.c:274
        #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
        #6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
        #7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

        --Type <return> to continue, or q <return> to quit--
        Thread 12 (Thread 0x7f4afeaf1700 (LWP 28416)):
        #0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
        #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
        #2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d8570) at beam/erl_threads.h:697
        #3 async_get (arg=0x7f4aff3d8570) at beam/erl_async.c:207
        #4 async_main (arg=0x7f4aff3d8570) at beam/erl_async.c:274
        #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
        #6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
        #7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

        Thread 11 (Thread 0x7f4afeb57700 (LWP 28413)):
        #0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
        #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
        #2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d8408) at beam/erl_threads.h:697
        #3 async_get (arg=0x7f4aff3d8408) at beam/erl_async.c:207
        #4 async_main (arg=0x7f4aff3d8408) at beam/erl_async.c:274
        #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
        #6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
        #7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

        Thread 10 (Thread 0x7f4afcbfe700 (LWP 28425)):
        #0 erts_smp_proc_raw_trylock__ (c_p=0x7f4aac2e15d0, ref=139958707146706) at beam/erl_process_lock.h:422
        #1 erts_smp_proc_lock__ (c_p=0x7f4aac2e15d0, ref=139958707146706) at beam/erl_process_lock.h:455
        #2 erts_smp_proc_lock (c_p=0x7f4aac2e15d0, ref=139958707146706) at beam/erl_process_lock.h:672
        #3 demonitor (c_p=0x7f4aac2e15d0, ref=139958707146706) at beam/bif.c:377
        #4 0x0000000000477c30 in demonitor_2 (A__p=0x7f4aac2e15d0, A_1=139958707146706, A_2=<value optimized out>) at beam/bif.c:505
        #5 0x0000000000521927 in process_main () at beam/beam_emu.c:2542
        #6 0x000000000048d496 in sched_thread_func (vesdp=0x7f4b00115200) at beam/erl_process.c:3782
        #7 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff316200a0) at pthread/ethread.c:106
        #8 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
        #9 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

        Thread 9 (Thread 0x7f4b00b7d700 (LWP 28400)):
        #0 0x000000397260e4ed in read () from /lib64/libpthread.so.0
        #1 0x000000000053810e in signal_dispatcher_thread_func (unused=<value optimized out>) at sys/unix/sys.c:2906
        #2 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff3161ffc0) at pthread/ethread.c:106
        #3 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
        #4 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

        Thread 8 (Thread 0x7f4b00ea9700 (LWP 28398)):
        #0 0x0000003971ede8b3 in select () from /lib64/libc.so.6
        #1 0x0000000000537ba0 in erts_sys_main_thread () at sys/unix/sys.c:3049
        #2 0x0000000000444761 in erl_start (argc=54, argv=<value optimized out>) at beam/erl_init.c:1441
        #3 0x0000000000426219 in main (argc=<value optimized out>, argv=<value optimized out>) at sys/unix/erl_main.c:29

        Thread 7 (Thread 0x7f4aae343700 (LWP 28979)):
        #0 0x0000003971eab15d in nanosleep () from /lib64/libc.so.6
        #1 0x0000003971edf124 in usleep () from /lib64/libc.so.6
        #2 0x00007f4af6371848 in v8::internal::ContextSwitcher::Run (this=0x7f4aa4030140) at src/v8threads.cc:457
        #3 0x00007f4af62f1580 in v8::internal::ThreadEntry (arg=0x7f4aa4030140) at src/platform-linux.cc:745
        #4 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
        #5 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

        Thread 6 (Thread 0x7f4afea69700 (LWP 28420)):
        #0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
        #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
        #2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d8750) at beam/erl_threads.h:697
        #3 async_get (arg=0x7f4aff3d8750) at beam/erl_async.c:207
        #4 async_main (arg=0x7f4aff3d8750) at beam/erl_async.c:274
        #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
        #6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
        #7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

        --Type <return> to continue, or q <return> to quit--
        Thread 5 (Thread 0x7f4afeb35700 (LWP 28414)):
        #0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
        #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
        #2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d8480) at beam/erl_threads.h:697
        #3 async_get (arg=0x7f4aff3d8480) at beam/erl_async.c:207
        #4 async_main (arg=0x7f4aff3d8480) at beam/erl_async.c:274
        #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
        #6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
        #7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

        Thread 4 (Thread 0x7f4afeb79700 (LWP 28412)):
        #0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
        #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
        #2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d8390) at beam/erl_threads.h:697
        #3 async_get (arg=0x7f4aff3d8390) at beam/erl_async.c:207
        #4 async_main (arg=0x7f4aff3d8390) at beam/erl_async.c:274
        #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
        #6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
        #7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

        Thread 3 (Thread 0x7f4afebbd700 (LWP 28410)):
        #0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
        #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
        #2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d82a0) at beam/erl_threads.h:697
        #3 async_get (arg=0x7f4aff3d82a0) at beam/erl_async.c:207
        #4 async_main (arg=0x7f4aff3d82a0) at beam/erl_async.c:274
        #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
        #6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
        #7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

        Thread 2 (Thread 0x7f4afebdf700 (LWP 28409)):
        #0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
        #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
        #2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d8228) at beam/erl_threads.h:697
        #3 async_get (arg=0x7f4aff3d8228) at beam/erl_async.c:207
        #4 async_main (arg=0x7f4aff3d8228) at beam/erl_async.c:274
        #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
        #6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
        #7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

        Thread 1 (Thread 0x7f4afec01700 (LWP 28408)):
        #0 0x000000000043ec6b in tree_delete (allctr=<value optimized out>, del=0x12c6a80, flags=<value optimized out>) at beam/erl_bestfit_alloc.c:435
        #1 0x0000000000437091 in mbc_free (allctr=0x12a86c0, p=<value optimized out>) at beam/erl_alloc_util.c:867
        #2 0x000000000043833f in do_erts_alcu_free (type=<value optimized out>, unused=<value optimized out>, p=0x12c6bb8) at beam/erl_alloc_util.c:2898
        #3 erts_alcu_free_thr_pref (type=<value optimized out>, unused=<value optimized out>, p=0x12c6bb8) at beam/erl_alloc_util.c:2949
        #4 0x000000000054ca48 in file_async_ready (e=0x12c3230, data=0x12c6bb8) at drivers/common/efile_drv.c:2117
        #5 0x0000000000480094 in async_ready (p=0x7f4afeca4e90, data=<value optimized out>) at beam/io.c:2445
        #6 0x00000000004e906f in async_main (arg=0x7f4aff3d81b0) at beam/erl_async.c:296
        #7 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
        #8 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
        #9 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

        Show
        karan Karan Kumar (Inactive) added a comment - GNU gdb (GDB) Red Hat Enterprise Linux (7.2-50.el6) Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later < http://gnu.org/licenses/gpl.html > This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: < http://www.gnu.org/software/gdb/bugs/ >... Reading symbols from /opt/couchbase/lib/erlang/erts-5.8.5/bin/beam.smp...done. [New Thread 28408] [New Thread 28409] [New Thread 28410] [New Thread 28412] [New Thread 28414] [New Thread 28420] [New Thread 28979] [New Thread 28398] [New Thread 28400] [New Thread 28425] [New Thread 28413] [New Thread 28416] [New Thread 28417] [New Thread 28433] [New Thread 28589] [New Thread 29574] [New Thread 28401] [New Thread 28405] [New Thread 28406] [New Thread 28424] [New Thread 28415] [New Thread 28419] [New Thread 29407] [New Thread 29576] [New Thread 28411] [New Thread 28418] [New Thread 28421] [New Thread 28403] [New Thread 28407] [New Thread 28423] [New Thread 28422] Reading symbols from /lib64/libutil.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libutil.so.1 Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libdl.so.2 Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libm.so.6 Reading symbols from /lib64/libncurses.so.5...(no debugging symbols found)...done. Loaded symbols for /lib64/libncurses.so.5 Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done. [Thread debugging using libthread_db enabled] Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/librt.so.1 Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /lib64/libtinfo.so.5...(no debugging symbols found)...done. Loaded symbols for /lib64/libtinfo.so.5 Reading symbols from /opt/couchbase/lib/erlang/lib/crypto-2.0.4/priv/lib/crypto.so...done. Loaded symbols for /opt/couchbase/lib/erlang/lib/crypto-2.0.4/priv/lib/crypto.so Reading symbols from /usr/lib64/libcrypto.so.6...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libcrypto.so.6 Reading symbols from /usr/lib64/libssl.so.6...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libssl.so.6 Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libz.so.1 Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libgssapi_krb5.so.2 Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols found)...done. Loaded symbols for /lib64/libkrb5.so.3 Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libcom_err.so.2 Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols found)...done. Loaded symbols for /lib64/libk5crypto.so.3 Reading symbols from /lib64/libkrb5support.so.0...(no debugging symbols found)...done. Loaded symbols for /lib64/libkrb5support.so.0 Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libkeyutils.so.1 Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libresolv.so.2 Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libselinux.so.1 Reading symbols from /opt/couchbase/lib/libv8.so...done. Loaded symbols for /opt/couchbase/lib/libv8.so Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libcrypt.so.1 Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libstdc++.so.6 Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libgcc_s.so.1 Reading symbols from /lib64/libfreebl3.so...(no debugging symbols found)...done. Loaded symbols for /lib64/libfreebl3.so Reading symbols from /opt/couchbase/lib/couchdb/erlang/lib/ejson-0.1.0/priv/ejson.so...done. Loaded symbols for /opt/couchbase/lib/couchdb/erlang/lib/ejson-0.1.0/priv/ejson.so Reading symbols from /opt/couchbase/lib/couchdb/erlang/lib/snappy-1.0.4/priv/snappy_nif.so...done. Loaded symbols for /opt/couchbase/lib/couchdb/erlang/lib/snappy-1.0.4/priv/snappy_nif.so Reading symbols from /opt/couchbase/lib/couchdb/erlang/lib/mapreduce-1.0/priv/mapreduce_nif.so...done. Loaded symbols for /opt/couchbase/lib/couchdb/erlang/lib/mapreduce-1.0/priv/mapreduce_nif.so Reading symbols from /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-5282953-git/priv/lib/couch_icu_driver.so...done. Loaded symbols for /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-5282953-git/priv/lib/couch_icu_driver.so Reading symbols from /opt/couchbase/lib/libicuuc.so.44...done. Loaded symbols for /opt/couchbase/lib/libicuuc.so.44 Reading symbols from /opt/couchbase/lib/libicudata.so.44...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/lib/libicudata.so.44 Reading symbols from /opt/couchbase/lib/libicui18n.so.44...done. Loaded symbols for /opt/couchbase/lib/libicui18n.so.44 Reading symbols from /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-5282953-git/priv/lib/couch_ejson_compare.so...done. Loaded symbols for /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-5282953-git/priv/lib/couch_ejson_compare.so Reading symbols from /opt/couchbase/lib/couchdb/erlang/lib/couch_view_parser-1.0/priv/couch_view_parser_nif.so...done. Loaded symbols for /opt/couchbase/lib/couchdb/erlang/lib/couch_view_parser-1.0/priv/couch_view_parser_nif.so Core was generated by `/opt/couchbase/lib/erlang/erts-5.8.5/bin/beam.smp -A 16 -sbt u -P 327680 -K tru'. Program terminated with signal 11, Segmentation fault. #0 0x000000000043ec6b in tree_delete (allctr=<value optimized out>, del=0x12c6a80, flags=<value optimized out>) at beam/erl_bestfit_alloc.c:435 435 beam/erl_bestfit_alloc.c: No such file or directory. in beam/erl_bestfit_alloc.c Missing separate debuginfos, use: debuginfo-install couchbase-server-2.0.0-1717.x86_64 (gdb) t a a bt Thread 31 (Thread 0x7f4afea01700 (LWP 28422)): #0 0x000000397260804d in pthread_join () from /lib64/libpthread.so.0 #1 0x00007f4af6371887 in v8::internal::ContextSwitcher::StopPreemption () at src/v8threads.cc:445 #2 0x00007f4af424b7a3 in destroyContext (ctx=0x7f4aac7b4f18) at mapreduce.cc:354 #3 0x00000000005138fb in nif_resource_dtor (bin=<value optimized out>) at beam/erl_nif.c:1217 #4 0x000000000048904b in erts_bin_free (offheap=<value optimized out>) at beam/erl_binary.h:303 #5 erts_cleanup_offheap (offheap=<value optimized out>) at beam/erl_message.c:174 #6 0x0000000000498a81 in delete_process (p=0x7f4aa286d568, pix_lock=<value optimized out>) at beam/erl_process.c:6621 #7 continue_exit_process (p=0x7f4aa286d568, pix_lock=<value optimized out>) at beam/erl_process.c:7663 #8 0x000000000051db94 in terminate_proc (c_p=0x7f4aa286d568, pc=<value optimized out>, reg=<value optimized out>, bf=<value optimized out>) at beam/beam_emu.c:5633 #9 handle_error (c_p=0x7f4aa286d568, pc=<value optimized out>, reg=<value optimized out>, bf=<value optimized out>) at beam/beam_emu.c:5508 #10 0x0000000000520cdf in process_main () at beam/beam_emu.c:3273 #11 0x000000000048d496 in sched_thread_func (vesdp=0x7f4b000eafc0) at beam/erl_process.c:3782 #12 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff316200a0) at pthread/ethread.c:106 #13 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0 #14 0x0000003971ee5ccd in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7f4afe000700 (LWP 28423)): #0 0x0000000000522f94 in process_main () at x86_64-unknown-linux-gnu/opt/smp/beam_hot.h:919 #1 0x000000000048d496 in sched_thread_func (vesdp=0x7f4b000f9080) at beam/erl_process.c:3782 #2 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff316200a0) at pthread/ethread.c:106 #3 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0 #4 0x0000003971ee5ccd in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7f4afec23700 (LWP 28407)): #0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350 #2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d8138) at beam/erl_threads.h:697 #3 async_get (arg=0x7f4aff3d8138) at beam/erl_async.c:207 #4 async_main (arg=0x7f4aff3d8138) at beam/erl_async.c:274 #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106 #6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0 #7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7f4affe69700 (LWP 28403)): #0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350 #2 0x0000000000462ad9 in erts_cnd_wait (unused=<value optimized out>) at beam/erl_threads.h:697 #3 erts_smp_cnd_wait (unused=<value optimized out>) at beam/erl_smp.h:472 #4 sys_msg_dispatcher_func (unused=<value optimized out>) at beam/erl_trace.c:3238 #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff316200d0) at pthread/ethread.c:106 #6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0 #7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7f4afea47700 (LWP 28421)): #0 0x000000397260f03d in waitpid () from /lib64/libpthread.so.0 #1 0x000000000053906f in child_waiter (unused=<value optimized out>) at sys/unix/sys.c:2821 #2 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620080) at pthread/ethread.c:106 #3 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0 #4 0x0000003971ee5ccd in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7f4afeaad700 (LWP 28418)): #0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350 #2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d8660) at beam/erl_threads.h:697 #3 async_get (arg=0x7f4aff3d8660) at beam/erl_async.c:207 #4 async_main (arg=0x7f4aff3d8660) at beam/erl_async.c:274 #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106 #6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0 #7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7f4afeb9b700 (LWP 28411)): -- Type <return> to continue, or q <return> to quit -- #0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350 #2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d8318) at beam/erl_threads.h:697 #3 async_get (arg=0x7f4aff3d8318) at beam/erl_async.c:207 #4 async_main (arg=0x7f4aff3d8318) at beam/erl_async.c:274 #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106 #6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0 #7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7f4aaed44700 (LWP 29576)): #0 0x0000003971eab15d in nanosleep () from /lib64/libc.so.6 #1 0x0000003971edf124 in usleep () from /lib64/libc.so.6 #2 0x00007f4af6371848 in v8::internal::ContextSwitcher::Run (this=0x7f4af814d8d0) at src/v8threads.cc:457 #3 0x00007f4af62f1580 in v8::internal::ThreadEntry (arg=0x7f4af814d8d0) at src/platform-linux.cc:745 #4 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003971ee5ccd in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7f4a94844700 (LWP 29407)): #0 0x0000003971eab15d in nanosleep () from /lib64/libc.so.6 #1 0x0000003971edf124 in usleep () from /lib64/libc.so.6 #2 0x00007f4af6371848 in v8::internal::ContextSwitcher::Run (this=0x7f4ab41b09d0) at src/v8threads.cc:457 #3 0x00007f4af62f1580 in v8::internal::ThreadEntry (arg=0x7f4ab41b09d0) at src/platform-linux.cc:745 #4 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003971ee5ccd in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7f4afea8b700 (LWP 28419)): #0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350 #2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d86d8) at beam/erl_threads.h:697 #3 async_get (arg=0x7f4aff3d86d8) at beam/erl_async.c:207 #4 async_main (arg=0x7f4aff3d86d8) at beam/erl_async.c:274 #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106 #6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0 #7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7f4afeb13700 (LWP 28415)): #0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350 #2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d84f8) at beam/erl_threads.h:697 #3 async_get (arg=0x7f4aff3d84f8) at beam/erl_async.c:207 #4 async_main (arg=0x7f4aff3d84f8) at beam/erl_async.c:274 #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106 #6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0 #7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7f4afd5ff700 (LWP 28424)): #0 do_erts_alcu_free (type=<value optimized out>, unused=<value optimized out>, p=0x7f4aaf0b8608) at beam/erl_alloc_util.c:2894 #1 erts_alcu_free_thr_pref (type=<value optimized out>, unused=<value optimized out>, p=0x7f4aaf0b8608) at beam/erl_alloc_util.c:2949 #2 0x0000000000489022 in erts_free (offheap=<value optimized out>) at beam/erl_alloc.h:226 #3 erts_bin_free (offheap=<value optimized out>) at beam/erl_binary.h:307 #4 erts_cleanup_offheap (offheap=<value optimized out>) at beam/erl_message.c:174 #5 0x0000000000498a81 in delete_process (p=0x7f4aac311310, pix_lock=<value optimized out>) at beam/erl_process.c:6621 #6 continue_exit_process (p=0x7f4aac311310, pix_lock=<value optimized out>) at beam/erl_process.c:7663 #7 0x000000000051db94 in terminate_proc (c_p=0x7f4aac311310, pc=<value optimized out>, reg=<value optimized out>, bf=<value optimized out>) at beam/beam_emu.c:5633 #8 handle_error (c_p=0x7f4aac311310, pc=<value optimized out>, reg=<value optimized out>, bf=<value optimized out>) at beam/beam_emu.c:5508 #9 0x0000000000520cdf in process_main () at beam/beam_emu.c:3273 #10 0x000000000048d496 in sched_thread_func (vesdp=0x7f4b00107140) at beam/erl_process.c:3782 #11 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff316200a0) at pthread/ethread.c:106 #12 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0 #13 0x0000003971ee5ccd in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7f4afec45700 (LWP 28406)): #0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 -- Type <return> to continue, or q <return> to quit -- #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350 #2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d80c0) at beam/erl_threads.h:697 #3 async_get (arg=0x7f4aff3d80c0) at beam/erl_async.c:207 #4 async_main (arg=0x7f4aff3d80c0) at beam/erl_async.c:274 #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106 #6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0 #7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7f4afec67700 (LWP 28405)): #0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350 #2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d8048) at beam/erl_threads.h:697 #3 async_get (arg=0x7f4aff3d8048) at beam/erl_async.c:207 #4 async_main (arg=0x7f4aff3d8048) at beam/erl_async.c:274 #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106 #6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0 #7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7f4b0017c700 (LWP 28401)): #0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350 #2 0x000000000046a1cf in erts_cnd_wait (unused=<value optimized out>) at beam/erl_threads.h:697 #3 erts_smp_cnd_wait (unused=<value optimized out>) at beam/erl_smp.h:472 #4 emergency_watchdog (unused=<value optimized out>) at beam/utils.c:3945 #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff316200d0) at pthread/ethread.c:106 #6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0 #7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f4a93e43700 (LWP 29574)): #0 0x0000003971eab15d in nanosleep () from /lib64/libc.so.6 #1 0x0000003971edf124 in usleep () from /lib64/libc.so.6 #2 0x00007f4af6371848 in v8::internal::ContextSwitcher::Run (this=0x7f4aa8091b00) at src/v8threads.cc:457 #3 0x00007f4af62f1580 in v8::internal::ThreadEntry (arg=0x7f4aa8091b00) at src/platform-linux.cc:745 #4 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003971ee5ccd in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f4aaed55700 (LWP 28589)): #0 0x000000397260d6c0 in sem_wait () from /lib64/libpthread.so.0 #1 0x00007f4af62f118d in v8::internal::LinuxSemaphore::Wait (this=0x7f4aa8009b10) at src/platform-linux.cc:869 #2 0x00007f4af630845c in v8::internal::RuntimeProfiler::WaitForSomeIsolateToEnterJS () at src/runtime-profiler.cc:386 #3 0x00007f4af62f2136 in v8::internal::SignalSender::Run (this=0x7f4aa8009470) at src/platform-linux.cc:1126 #4 0x00007f4af62f1580 in v8::internal::ThreadEntry (arg=0x7f4aa8009470) at src/platform-linux.cc:745 #5 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0 #6 0x0000003971ee5ccd in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f4aaffff700 (LWP 28433)): #0 0x0000003971eab15d in nanosleep () from /lib64/libc.so.6 #1 0x0000003971edf124 in usleep () from /lib64/libc.so.6 #2 0x00007f4af4249397 in terminatorLoop (args=<value optimized out>) at mapreduce_nif.cc:445 #3 0x0000000000584d89 in thr_wrapper (vtwd=0x7f4afea00ae0) at pthread/ethread.c:106 #4 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003971ee5ccd in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f4afeacf700 (LWP 28417)): #0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350 #2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d85e8) at beam/erl_threads.h:697 #3 async_get (arg=0x7f4aff3d85e8) at beam/erl_async.c:207 #4 async_main (arg=0x7f4aff3d85e8) at beam/erl_async.c:274 #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106 #6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0 #7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6 -- Type <return> to continue, or q <return> to quit -- Thread 12 (Thread 0x7f4afeaf1700 (LWP 28416)): #0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350 #2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d8570) at beam/erl_threads.h:697 #3 async_get (arg=0x7f4aff3d8570) at beam/erl_async.c:207 #4 async_main (arg=0x7f4aff3d8570) at beam/erl_async.c:274 #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106 #6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0 #7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f4afeb57700 (LWP 28413)): #0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350 #2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d8408) at beam/erl_threads.h:697 #3 async_get (arg=0x7f4aff3d8408) at beam/erl_async.c:207 #4 async_main (arg=0x7f4aff3d8408) at beam/erl_async.c:274 #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106 #6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0 #7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f4afcbfe700 (LWP 28425)): #0 erts_smp_proc_raw_trylock__ (c_p=0x7f4aac2e15d0, ref=139958707146706) at beam/erl_process_lock.h:422 #1 erts_smp_proc_lock__ (c_p=0x7f4aac2e15d0, ref=139958707146706) at beam/erl_process_lock.h:455 #2 erts_smp_proc_lock (c_p=0x7f4aac2e15d0, ref=139958707146706) at beam/erl_process_lock.h:672 #3 demonitor (c_p=0x7f4aac2e15d0, ref=139958707146706) at beam/bif.c:377 #4 0x0000000000477c30 in demonitor_2 (A__p=0x7f4aac2e15d0, A_1=139958707146706, A_2=<value optimized out>) at beam/bif.c:505 #5 0x0000000000521927 in process_main () at beam/beam_emu.c:2542 #6 0x000000000048d496 in sched_thread_func (vesdp=0x7f4b00115200) at beam/erl_process.c:3782 #7 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff316200a0) at pthread/ethread.c:106 #8 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0 #9 0x0000003971ee5ccd in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f4b00b7d700 (LWP 28400)): #0 0x000000397260e4ed in read () from /lib64/libpthread.so.0 #1 0x000000000053810e in signal_dispatcher_thread_func (unused=<value optimized out>) at sys/unix/sys.c:2906 #2 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff3161ffc0) at pthread/ethread.c:106 #3 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0 #4 0x0000003971ee5ccd in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f4b00ea9700 (LWP 28398)): #0 0x0000003971ede8b3 in select () from /lib64/libc.so.6 #1 0x0000000000537ba0 in erts_sys_main_thread () at sys/unix/sys.c:3049 #2 0x0000000000444761 in erl_start (argc=54, argv=<value optimized out>) at beam/erl_init.c:1441 #3 0x0000000000426219 in main (argc=<value optimized out>, argv=<value optimized out>) at sys/unix/erl_main.c:29 Thread 7 (Thread 0x7f4aae343700 (LWP 28979)): #0 0x0000003971eab15d in nanosleep () from /lib64/libc.so.6 #1 0x0000003971edf124 in usleep () from /lib64/libc.so.6 #2 0x00007f4af6371848 in v8::internal::ContextSwitcher::Run (this=0x7f4aa4030140) at src/v8threads.cc:457 #3 0x00007f4af62f1580 in v8::internal::ThreadEntry (arg=0x7f4aa4030140) at src/platform-linux.cc:745 #4 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003971ee5ccd in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f4afea69700 (LWP 28420)): #0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350 #2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d8750) at beam/erl_threads.h:697 #3 async_get (arg=0x7f4aff3d8750) at beam/erl_async.c:207 #4 async_main (arg=0x7f4aff3d8750) at beam/erl_async.c:274 #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106 #6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0 #7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6 -- Type <return> to continue, or q <return> to quit -- Thread 5 (Thread 0x7f4afeb35700 (LWP 28414)): #0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350 #2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d8480) at beam/erl_threads.h:697 #3 async_get (arg=0x7f4aff3d8480) at beam/erl_async.c:207 #4 async_main (arg=0x7f4aff3d8480) at beam/erl_async.c:274 #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106 #6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0 #7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f4afeb79700 (LWP 28412)): #0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350 #2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d8390) at beam/erl_threads.h:697 #3 async_get (arg=0x7f4aff3d8390) at beam/erl_async.c:207 #4 async_main (arg=0x7f4aff3d8390) at beam/erl_async.c:274 #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106 #6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0 #7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f4afebbd700 (LWP 28410)): #0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350 #2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d82a0) at beam/erl_threads.h:697 #3 async_get (arg=0x7f4aff3d82a0) at beam/erl_async.c:207 #4 async_main (arg=0x7f4aff3d82a0) at beam/erl_async.c:274 #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106 #6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0 #7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f4afebdf700 (LWP 28409)): #0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350 #2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d8228) at beam/erl_threads.h:697 #3 async_get (arg=0x7f4aff3d8228) at beam/erl_async.c:207 #4 async_main (arg=0x7f4aff3d8228) at beam/erl_async.c:274 #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106 #6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0 #7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f4afec01700 (LWP 28408)): #0 0x000000000043ec6b in tree_delete (allctr=<value optimized out>, del=0x12c6a80, flags=<value optimized out>) at beam/erl_bestfit_alloc.c:435 #1 0x0000000000437091 in mbc_free (allctr=0x12a86c0, p=<value optimized out>) at beam/erl_alloc_util.c:867 #2 0x000000000043833f in do_erts_alcu_free (type=<value optimized out>, unused=<value optimized out>, p=0x12c6bb8) at beam/erl_alloc_util.c:2898 #3 erts_alcu_free_thr_pref (type=<value optimized out>, unused=<value optimized out>, p=0x12c6bb8) at beam/erl_alloc_util.c:2949 #4 0x000000000054ca48 in file_async_ready (e=0x12c3230, data=0x12c6bb8) at drivers/common/efile_drv.c:2117 #5 0x0000000000480094 in async_ready (p=0x7f4afeca4e90, data=<value optimized out>) at beam/io.c:2445 #6 0x00000000004e906f in async_main (arg=0x7f4aff3d81b0) at beam/erl_async.c:296 #7 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106 #8 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0 #9 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
        karan Karan Kumar (Inactive) made changes -
        Assignee Thuan Nguyen [ thuan ] Aleksey Kondratenko [ alkondratenko ]
        Show
        karan Karan Kumar (Inactive) added a comment - https://friendpaste.com/6ctSNIR61xOtG8HSieJ4qA
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        Another beam crash in some memory management code in beam. Perhaps duplicate of some other crash we have

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - Another beam crash in some memory management code in beam. Perhaps duplicate of some other crash we have
        alkondratenko Aleksey Kondratenko (Inactive) made changes -
        Assignee Aleksey Kondratenko [ alkondratenko ] Damien Katz [ damien ]
        karan Karan Kumar (Inactive) made changes -
        Assignee Damien Katz [ damien ] Aleksey Kondratenko [ alkondratenko ]
        Show
        thuan Thuan Nguyen added a comment - Raw erlang core dump is here https://s3.amazonaws.com/packages.couchbase/core-dump/orange/201209/core.beam.smp.28398.gz
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        Karan, what you want me to do with that?

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - Karan, what you want me to do with that?
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        I've heard you were looking for core dump?

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - I've heard you were looking for core dump?
        alkondratenko Aleksey Kondratenko (Inactive) made changes -
        Assignee Aleksey Kondratenko [ alkondratenko ] Damien Katz [ damien ]
        Hide
        karan Karan Kumar (Inactive) added a comment -

        Ahh.... If we have duplicate of this bug.. somewhere we can close it..
        But the segfault's of any component IMHO are quite severe

        Show
        karan Karan Kumar (Inactive) added a comment - Ahh.... If we have duplicate of this bug.. somewhere we can close it.. But the segfault's of any component IMHO are quite severe
        farshid Farshid Ghods (Inactive) made changes -
        Summary [longevity] Newly added node, went down due to beam.smp segfaulted [longevity] segmentation fault when rebalancing a new node to a cluster with 2 buckets and 1 design doc per bucket
        farshid Farshid Ghods (Inactive) made changes -
        Summary [longevity] segmentation fault when rebalancing a new node to a cluster with 2 buckets and 1 design doc per bucket [longevity] beam.smp ( erl_bestfit_alloc.c:435 ) segmentation fault when rebalancing a new node to a cluster with 2 buckets and 1 design doc per bucket
        farshid Farshid Ghods (Inactive) made changes -
        Summary [longevity] beam.smp ( erl_bestfit_alloc.c:435 ) segmentation fault when rebalancing a new node to a cluster with 2 buckets and 1 design doc per bucket beam.smp ( erl_bestfit_alloc.c:435 ) segmentation fault when rebalancing a new node to a cluster with 2 buckets and 1 design doc per bucket
        Hide
        karan Karan Kumar (Inactive) added a comment -

        Another beam.smp segfault:-

        #0 right_rotate (allctr=<value optimized out>, del=<value optimized out>, flags=<value optimized out>) at beam/erl_bestfit_alloc.c:252
        252 beam/erl_bestfit_alloc.c: No such file or directory.
        in beam/erl_bestfit_alloc.c
        Missing separate debuginfos, use: debuginfo-install couchbase-server-2.0.0-1728.x86_64
        (gdb) t a a bt

        Thread 28 (Thread 0x7f4b54ff3700 (LWP 6632)):
        #0 0x000000344ee0ebad in fsync () from /lib64/libpthread.so.0
        #1 0x000000000053b8db in efile_fsync (errInfo=0xa9744c, fd=<value optimized out>) at drivers/unix/unix_efile.c:806
        #2 0x0000000000550e18 in invoke_fsync (data=0xa97420) at drivers/common/efile_drv.c:940
        #3 0x00000000004e8f55 in async_main (arg=0x7f4b557a8138) at beam/erl_async.c:281
        #4 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
        #5 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
        #6 0x000000344e6e5ccd in clone () from /lib64/libc.so.6

        Thread 27 (Thread 0x7f4b54dd1700 (LWP 6647)):
        #0 0x000000344ee0dff4 in __lll_lock_wait () from /lib64/libpthread.so.0
        #1 0x000000344ee09328 in _L_lock_854 () from /lib64/libpthread.so.0
        #2 0x000000344ee091f7 in pthread_mutex_lock () from /lib64/libpthread.so.0
        #3 0x000000000043811f in ethr_mutex_lock (type=<value optimized out>, extra=<value optimized out>, size=280) at ../include/internal/ethr_mutex.h:589
        #4 erts_mtx_lock (type=<value optimized out>, extra=<value optimized out>, size=280) at beam/erl_threads.h:622
        #5 erts_alcu_alloc_thr_pref (type=<value optimized out>, extra=<value optimized out>, size=280) at beam/erl_alloc_util.c:2864
        #6 0x000000000054fa5b in file_outputv (e=0xa8fb90, ev=0x7f4b54dd0c10) at drivers/common/efile_drv.c:2563
        #7 0x0000000000483c78 in erts_write_to_port (caller_id=<value optimized out>, p=0x7f4b54dd0a10, list=139960099274753) at beam/io.c:1159
        #8 0x00000000004f5301 in do_port_command (A__p=0x7f4b029c3060, A_1=638167, A_2=139961559834186, A_3=<value optimized out>, flags=0) at beam/erl_bif_port.c:181
        #9 0x0000000000521927 in process_main () at beam/beam_emu.c:2542
        #10 0x000000000048d496 in sched_thread_func (vesdp=0x7f4b564bafc0) at beam/erl_process.c:3782
        #11 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5fc0) at pthread/ethread.c:106
        #12 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
        #13 0x000000344e6e5ccd in clone () from /lib64/libc.so.6

        Thread 26 (Thread 0x7f4b539cf700 (LWP 6649)):
        #0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
        #1 0x0000000000585463 in wait__ (e=0xa19c60) at pthread/ethr_event.c:185
        #2 ethr_event_wait (e=0xa19c60) at pthread/ethr_event.c:218
        #3 0x000000000049821d in erts_tse_wait (fcalls=<value optimized out>, esdp=0x7f4b564d7140, rq=0x7f4b564ba7c0) at beam/erl_threads.h:1809
        #4 scheduler_wait (fcalls=<value optimized out>, esdp=0x7f4b564d7140, rq=0x7f4b564ba7c0) at beam/erl_process.c:1076
        #5 0x000000000049a248 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:5421
        #6 0x000000000051e65f in process_main () at beam/beam_emu.c:1225
        #7 0x000000000048d496 in sched_thread_func (vesdp=0x7f4b564d7140) at beam/erl_process.c:3782
        #8 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5fc0) at pthread/ethread.c:106
        #9 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
        #10 0x000000344e6e5ccd in clone () from /lib64/libc.so.6

        Thread 25 (Thread 0x7f4b543d0700 (LWP 6648)):
        #0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
        #1 0x0000000000585463 in wait__ (e=0xa19ba0) at pthread/ethr_event.c:185
        #2 ethr_event_wait (e=0xa19ba0) at pthread/ethr_event.c:218
        #3 0x000000000049821d in erts_tse_wait (fcalls=<value optimized out>, esdp=0x7f4b564c9080, rq=0x7f4b564ba5c0) at beam/erl_threads.h:1809
        #4 scheduler_wait (fcalls=<value optimized out>, esdp=0x7f4b564c9080, rq=0x7f4b564ba5c0) at beam/erl_process.c:1076
        #5 0x000000000049a248 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:5421
        #6 0x000000000051e65f in process_main () at beam/beam_emu.c:1225
        #7 0x000000000048d496 in sched_thread_func (vesdp=0x7f4b564c9080) at beam/erl_process.c:3782
        #8 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5fc0) at pthread/ethread.c:106
        #9 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
        #10 0x000000344e6e5ccd in clone () from /lib64/libc.so.6

        Thread 24 (Thread 0x7f4b52fce700 (LWP 6650)):
        #0 0x000000344e6e62c3 in epoll_wait () from /lib64/libc.so.6
        #1 0x000000000053de47 in check_fd_events (ps=0x7f4b570012c8, pr=0x7f4b52fcd360, len=0x7f4b52fcdb7c, utvp=0x7f4b52fcdb60) at sys/common/erl_poll.c:1918
        #2 erts_poll_wait_kp (ps=0x7f4b570012c8, pr=0x7f4b52fcd360, len=0x7f4b52fcdb7c, utvp=0x7f4b52fcdb60) at sys/common/erl_poll.c:2053
        #3 0x0000000000540efe in erts_check_io_kp (do_wait=<value optimized out>) at sys/common/erl_check_io.c:1161
        #4 0x0000000000498551 in scheduler_wait (fcalls=<value optimized out>, esdp=0x7f4b564e5200, rq=0x7f4b564ba9c0) at beam/erl_process.c:1221
        #5 0x000000000049a248 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:5421
        #6 0x000000000051e65f in process_main () at beam/beam_emu.c:1225
        #7 0x000000000048d496 in sched_thread_func (vesdp=0x7f4b564e5200) at beam/erl_process.c:3782
        #8 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5fc0) at pthread/ethread.c:106
        #9 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
        --Type <return> to continue, or q <return> to quit--
        #10 0x000000344e6e5ccd in clone () from /lib64/libc.so.6

        Thread 23 (Thread 0x7f4afdee0700 (LWP 7076)):
        #0 0x000000344ee0d6c0 in sem_wait () from /lib64/libpthread.so.0
        #1 0x00007f4b50a6718d in v8::internal::LinuxSemaphore::Wait (this=0x7f4b08009b10) at src/platform-linux.cc:869
        #2 0x00007f4b50a7e45c in v8::internal::RuntimeProfiler::WaitForSomeIsolateToEnterJS () at src/runtime-profiler.cc:386
        #3 0x00007f4b50a68136 in v8::internal::SignalSender::Run (this=0x7f4af80019c0) at src/platform-linux.cc:1126
        #4 0x00007f4b50a67580 in v8::internal::ThreadEntry (arg=0x7f4af80019c0) at src/platform-linux.cc:745
        #5 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
        #6 0x000000344e6e5ccd in clone () from /lib64/libc.so.6

        Thread 22 (Thread 0x7f4b54f27700 (LWP 6638)):
        #0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
        #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
        #2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a8408) at beam/erl_threads.h:697
        #3 async_get (arg=0x7f4b557a8408) at beam/erl_async.c:207
        #4 async_main (arg=0x7f4b557a8408) at beam/erl_async.c:274
        #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
        #6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
        #7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6

        Thread 21 (Thread 0x7f4b5654c700 (LWP 6628)):
        #0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
        #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
        #2 0x000000000046a1cf in erts_cnd_wait (unused=<value optimized out>) at beam/erl_threads.h:697
        #3 erts_smp_cnd_wait (unused=<value optimized out>) at beam/erl_smp.h:472
        #4 emergency_watchdog (unused=<value optimized out>) at beam/utils.c:3945
        #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5ff0) at pthread/ethread.c:106
        #6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
        #7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6

        Thread 20 (Thread 0x7f4b54e9f700 (LWP 6642)):
        #0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
        #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
        #2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a85e8) at beam/erl_threads.h:697
        #3 async_get (arg=0x7f4b557a85e8) at beam/erl_async.c:207
        #4 async_main (arg=0x7f4b557a85e8) at beam/erl_async.c:274
        #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
        #6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
        #7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6

        Thread 19 (Thread 0x7f4b54faf700 (LWP 6634)):
        #0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
        #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
        #2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a8228) at beam/erl_threads.h:697
        #3 async_get (arg=0x7f4b557a8228) at beam/erl_async.c:207
        #4 async_main (arg=0x7f4b557a8228) at beam/erl_async.c:274
        #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
        #6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
        #7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6

        Thread 18 (Thread 0x7f4b54ee3700 (LWP 6640)):
        #0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
        #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
        #2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a84f8) at beam/erl_threads.h:697
        #3 async_get (arg=0x7f4b557a84f8) at beam/erl_async.c:207
        #4 async_main (arg=0x7f4b557a84f8) at beam/erl_async.c:274
        #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
        #6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
        #7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6

        Thread 17 (Thread 0x7f4b56239700 (LWP 6629)):
        #0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
        --Type <return> to continue, or q <return> to quit--
        #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
        #2 0x0000000000462ad9 in erts_cnd_wait (unused=<value optimized out>) at beam/erl_threads.h:697
        #3 erts_smp_cnd_wait (unused=<value optimized out>) at beam/erl_smp.h:472
        #4 sys_msg_dispatcher_func (unused=<value optimized out>) at beam/erl_trace.c:3238
        #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5ff0) at pthread/ethread.c:106
        #6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
        #7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6

        Thread 16 (Thread 0x7f4b54f8d700 (LWP 6635)):
        #0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
        #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
        #2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a82a0) at beam/erl_threads.h:697
        #3 async_get (arg=0x7f4b557a82a0) at beam/erl_async.c:207
        #4 async_main (arg=0x7f4b557a82a0) at beam/erl_async.c:274
        #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
        #6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
        #7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6

        Thread 15 (Thread 0x7f4b0062c700 (LWP 6660)):
        #0 0x000000344e6ab15d in nanosleep () from /lib64/libc.so.6
        #1 0x000000344e6df124 in usleep () from /lib64/libc.so.6
        #2 0x00007f4b00631397 in terminatorLoop (args=<value optimized out>) at mapreduce_nif.cc:445
        #3 0x0000000000584d89 in thr_wrapper (vtwd=0x7f4b54dd0ae0) at pthread/ethread.c:106
        #4 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
        #5 0x000000344e6e5ccd in clone () from /lib64/libc.so.6

        Thread 14 (Thread 0x7f4b54f05700 (LWP 6639)):
        #0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
        #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
        #2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a8480) at beam/erl_threads.h:697
        #3 async_get (arg=0x7f4b557a8480) at beam/erl_async.c:207
        #4 async_main (arg=0x7f4b557a8480) at beam/erl_async.c:274
        #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
        #6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
        #7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6

        Thread 13 (Thread 0x7f4b55015700 (LWP 6631)):
        #0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
        #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
        #2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a80c0) at beam/erl_threads.h:697
        #3 async_get (arg=0x7f4b557a80c0) at beam/erl_async.c:207
        #4 async_main (arg=0x7f4b557a80c0) at beam/erl_async.c:274
        #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
        #6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
        #7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6

        Thread 12 (Thread 0x7f4af3171700 (LWP 7145)):
        #0 0x000000344e6ab15d in nanosleep () from /lib64/libc.so.6
        #1 0x000000344e6df124 in usleep () from /lib64/libc.so.6
        #2 0x00007f4b50ae7848 in v8::internal::ContextSwitcher::Run (this=0x7f4b0414ace0) at src/v8threads.cc:457
        #3 0x00007f4b50a67580 in v8::internal::ThreadEntry (arg=0x7f4b0414ace0) at src/platform-linux.cc:745
        #4 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
        #5 0x000000344e6e5ccd in clone () from /lib64/libc.so.6

        Thread 11 (Thread 0x7f4b56f4d700 (LWP 6627)):
        #0 0x000000344ee0e4ed in read () from /lib64/libpthread.so.0
        #1 0x000000000053810e in signal_dispatcher_thread_func (unused=<value optimized out>) at sys/unix/sys.c:2906
        #2 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5ee0) at pthread/ethread.c:106
        #3 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
        #4 0x000000344e6e5ccd in clone () from /lib64/libc.so.6

        Thread 10 (Thread 0x7f4b54ec1700 (LWP 6641)):
        #0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
        --Type <return> to continue, or q <return> to quit--
        #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
        #2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a8570) at beam/erl_threads.h:697
        #3 async_get (arg=0x7f4b557a8570) at beam/erl_async.c:207
        #4 async_main (arg=0x7f4b557a8570) at beam/erl_async.c:274
        #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
        #6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
        #7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6

        Thread 9 (Thread 0x7f4b54f49700 (LWP 6637)):
        #0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
        #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
        #2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a8390) at beam/erl_threads.h:697
        #3 async_get (arg=0x7f4b557a8390) at beam/erl_async.c:207
        #4 async_main (arg=0x7f4b557a8390) at beam/erl_async.c:274
        #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
        #6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
        #7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6

        Thread 8 (Thread 0x7f4b57279700 (LWP 6626)):
        #0 0x000000344e6de8b3 in select () from /lib64/libc.so.6
        #1 0x0000000000537ba0 in erts_sys_main_thread () at sys/unix/sys.c:3049
        #2 0x0000000000444761 in erl_start (argc=54, argv=<value optimized out>) at beam/erl_init.c:1441
        #3 0x0000000000426219 in main (argc=<value optimized out>, argv=<value optimized out>) at sys/unix/erl_main.c:29

        Thread 7 (Thread 0x7f4b54f6b700 (LWP 6636)):
        #0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
        #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
        #2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a8318) at beam/erl_threads.h:697
        #3 async_get (arg=0x7f4b557a8318) at beam/erl_async.c:207
        #4 async_main (arg=0x7f4b557a8318) at beam/erl_async.c:274
        #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
        #6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
        #7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6

        Thread 6 (Thread 0x7f4b54fd1700 (LWP 6633)):
        #0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
        #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
        #2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a81b0) at beam/erl_threads.h:697
        #3 async_get (arg=0x7f4b557a81b0) at beam/erl_async.c:207
        #4 async_main (arg=0x7f4b557a81b0) at beam/erl_async.c:274
        #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
        #6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
        #7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6

        Thread 5 (Thread 0x7f4b54e17700 (LWP 6646)):
        #0 0x000000344ee0f03d in waitpid () from /lib64/libpthread.so.0
        #1 0x000000000053906f in child_waiter (unused=<value optimized out>) at sys/unix/sys.c:2821
        #2 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5fa0) at pthread/ethread.c:106
        #3 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
        #4 0x000000344e6e5ccd in clone () from /lib64/libc.so.6

        Thread 4 (Thread 0x7f4b55037700 (LWP 6630)):
        #0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
        #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
        #2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a8048) at beam/erl_threads.h:697
        #3 async_get (arg=0x7f4b557a8048) at beam/erl_async.c:207
        #4 async_main (arg=0x7f4b557a8048) at beam/erl_async.c:274
        #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
        #6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
        #7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6

        Thread 3 (Thread 0x7f4b54e5b700 (LWP 6644)):
        #0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
        --Type <return> to continue, or q <return> to quit--
        #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
        #2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a86d8) at beam/erl_threads.h:697
        #3 async_get (arg=0x7f4b557a86d8) at beam/erl_async.c:207
        #4 async_main (arg=0x7f4b557a86d8) at beam/erl_async.c:274
        #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
        #6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
        #7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6

        Thread 2 (Thread 0x7f4b54e39700 (LWP 6645)):
        #0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
        #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
        #2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a8750) at beam/erl_threads.h:697
        #3 async_get (arg=0x7f4b557a8750) at beam/erl_async.c:207
        #4 async_main (arg=0x7f4b557a8750) at beam/erl_async.c:274
        #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
        #6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
        #7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6

        Thread 1 (Thread 0x7f4b54e7d700 (LWP 6643)):
        #0 right_rotate (allctr=<value optimized out>, del=<value optimized out>, flags=<value optimized out>) at beam/erl_bestfit_alloc.c:252
        #1 tree_delete (allctr=<value optimized out>, del=<value optimized out>, flags=<value optimized out>) at beam/erl_bestfit_alloc.c:496
        #2 0x00000000004370db in mbc_free (allctr=0xa7c2c0, p=<value optimized out>) at beam/erl_alloc_util.c:883
        #3 0x000000000043833f in do_erts_alcu_free (type=<value optimized out>, unused=<value optimized out>, p=0xa87a38) at beam/erl_alloc_util.c:2898
        #4 erts_alcu_free_thr_pref (type=<value optimized out>, unused=<value optimized out>, p=0xa87a38) at beam/erl_alloc_util.c:2949
        #5 0x000000000054cac0 in file_async_ready (e=0xa84238, data=0xa87a38) at drivers/common/efile_drv.c:2091
        #6 0x0000000000480094 in async_ready (p=0x7f4b553521e0, data=<value optimized out>) at beam/io.c:2445
        #7 0x00000000004e906f in async_main (arg=0x7f4b557a8660) at beam/erl_async.c:296
        #8 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
        #9 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
        #10 0x000000344e6e5ccd in clone () from /lib64/libc.so.6

        Show
        karan Karan Kumar (Inactive) added a comment - Another beam.smp segfault:- #0 right_rotate (allctr=<value optimized out>, del=<value optimized out>, flags=<value optimized out>) at beam/erl_bestfit_alloc.c:252 252 beam/erl_bestfit_alloc.c: No such file or directory. in beam/erl_bestfit_alloc.c Missing separate debuginfos, use: debuginfo-install couchbase-server-2.0.0-1728.x86_64 (gdb) t a a bt Thread 28 (Thread 0x7f4b54ff3700 (LWP 6632)): #0 0x000000344ee0ebad in fsync () from /lib64/libpthread.so.0 #1 0x000000000053b8db in efile_fsync (errInfo=0xa9744c, fd=<value optimized out>) at drivers/unix/unix_efile.c:806 #2 0x0000000000550e18 in invoke_fsync (data=0xa97420) at drivers/common/efile_drv.c:940 #3 0x00000000004e8f55 in async_main (arg=0x7f4b557a8138) at beam/erl_async.c:281 #4 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106 #5 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0 #6 0x000000344e6e5ccd in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7f4b54dd1700 (LWP 6647)): #0 0x000000344ee0dff4 in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x000000344ee09328 in _L_lock_854 () from /lib64/libpthread.so.0 #2 0x000000344ee091f7 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x000000000043811f in ethr_mutex_lock (type=<value optimized out>, extra=<value optimized out>, size=280) at ../include/internal/ethr_mutex.h:589 #4 erts_mtx_lock (type=<value optimized out>, extra=<value optimized out>, size=280) at beam/erl_threads.h:622 #5 erts_alcu_alloc_thr_pref (type=<value optimized out>, extra=<value optimized out>, size=280) at beam/erl_alloc_util.c:2864 #6 0x000000000054fa5b in file_outputv (e=0xa8fb90, ev=0x7f4b54dd0c10) at drivers/common/efile_drv.c:2563 #7 0x0000000000483c78 in erts_write_to_port (caller_id=<value optimized out>, p=0x7f4b54dd0a10, list=139960099274753) at beam/io.c:1159 #8 0x00000000004f5301 in do_port_command (A__p=0x7f4b029c3060, A_1=638167, A_2=139961559834186, A_3=<value optimized out>, flags=0) at beam/erl_bif_port.c:181 #9 0x0000000000521927 in process_main () at beam/beam_emu.c:2542 #10 0x000000000048d496 in sched_thread_func (vesdp=0x7f4b564bafc0) at beam/erl_process.c:3782 #11 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5fc0) at pthread/ethread.c:106 #12 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0 #13 0x000000344e6e5ccd in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7f4b539cf700 (LWP 6649)): #0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000585463 in wait__ (e=0xa19c60) at pthread/ethr_event.c:185 #2 ethr_event_wait (e=0xa19c60) at pthread/ethr_event.c:218 #3 0x000000000049821d in erts_tse_wait (fcalls=<value optimized out>, esdp=0x7f4b564d7140, rq=0x7f4b564ba7c0) at beam/erl_threads.h:1809 #4 scheduler_wait (fcalls=<value optimized out>, esdp=0x7f4b564d7140, rq=0x7f4b564ba7c0) at beam/erl_process.c:1076 #5 0x000000000049a248 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:5421 #6 0x000000000051e65f in process_main () at beam/beam_emu.c:1225 #7 0x000000000048d496 in sched_thread_func (vesdp=0x7f4b564d7140) at beam/erl_process.c:3782 #8 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5fc0) at pthread/ethread.c:106 #9 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0 #10 0x000000344e6e5ccd in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7f4b543d0700 (LWP 6648)): #0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000585463 in wait__ (e=0xa19ba0) at pthread/ethr_event.c:185 #2 ethr_event_wait (e=0xa19ba0) at pthread/ethr_event.c:218 #3 0x000000000049821d in erts_tse_wait (fcalls=<value optimized out>, esdp=0x7f4b564c9080, rq=0x7f4b564ba5c0) at beam/erl_threads.h:1809 #4 scheduler_wait (fcalls=<value optimized out>, esdp=0x7f4b564c9080, rq=0x7f4b564ba5c0) at beam/erl_process.c:1076 #5 0x000000000049a248 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:5421 #6 0x000000000051e65f in process_main () at beam/beam_emu.c:1225 #7 0x000000000048d496 in sched_thread_func (vesdp=0x7f4b564c9080) at beam/erl_process.c:3782 #8 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5fc0) at pthread/ethread.c:106 #9 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0 #10 0x000000344e6e5ccd in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7f4b52fce700 (LWP 6650)): #0 0x000000344e6e62c3 in epoll_wait () from /lib64/libc.so.6 #1 0x000000000053de47 in check_fd_events (ps=0x7f4b570012c8, pr=0x7f4b52fcd360, len=0x7f4b52fcdb7c, utvp=0x7f4b52fcdb60) at sys/common/erl_poll.c:1918 #2 erts_poll_wait_kp (ps=0x7f4b570012c8, pr=0x7f4b52fcd360, len=0x7f4b52fcdb7c, utvp=0x7f4b52fcdb60) at sys/common/erl_poll.c:2053 #3 0x0000000000540efe in erts_check_io_kp (do_wait=<value optimized out>) at sys/common/erl_check_io.c:1161 #4 0x0000000000498551 in scheduler_wait (fcalls=<value optimized out>, esdp=0x7f4b564e5200, rq=0x7f4b564ba9c0) at beam/erl_process.c:1221 #5 0x000000000049a248 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:5421 #6 0x000000000051e65f in process_main () at beam/beam_emu.c:1225 #7 0x000000000048d496 in sched_thread_func (vesdp=0x7f4b564e5200) at beam/erl_process.c:3782 #8 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5fc0) at pthread/ethread.c:106 #9 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0 -- Type <return> to continue, or q <return> to quit -- #10 0x000000344e6e5ccd in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7f4afdee0700 (LWP 7076)): #0 0x000000344ee0d6c0 in sem_wait () from /lib64/libpthread.so.0 #1 0x00007f4b50a6718d in v8::internal::LinuxSemaphore::Wait (this=0x7f4b08009b10) at src/platform-linux.cc:869 #2 0x00007f4b50a7e45c in v8::internal::RuntimeProfiler::WaitForSomeIsolateToEnterJS () at src/runtime-profiler.cc:386 #3 0x00007f4b50a68136 in v8::internal::SignalSender::Run (this=0x7f4af80019c0) at src/platform-linux.cc:1126 #4 0x00007f4b50a67580 in v8::internal::ThreadEntry (arg=0x7f4af80019c0) at src/platform-linux.cc:745 #5 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0 #6 0x000000344e6e5ccd in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7f4b54f27700 (LWP 6638)): #0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350 #2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a8408) at beam/erl_threads.h:697 #3 async_get (arg=0x7f4b557a8408) at beam/erl_async.c:207 #4 async_main (arg=0x7f4b557a8408) at beam/erl_async.c:274 #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106 #6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0 #7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7f4b5654c700 (LWP 6628)): #0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350 #2 0x000000000046a1cf in erts_cnd_wait (unused=<value optimized out>) at beam/erl_threads.h:697 #3 erts_smp_cnd_wait (unused=<value optimized out>) at beam/erl_smp.h:472 #4 emergency_watchdog (unused=<value optimized out>) at beam/utils.c:3945 #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5ff0) at pthread/ethread.c:106 #6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0 #7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7f4b54e9f700 (LWP 6642)): #0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350 #2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a85e8) at beam/erl_threads.h:697 #3 async_get (arg=0x7f4b557a85e8) at beam/erl_async.c:207 #4 async_main (arg=0x7f4b557a85e8) at beam/erl_async.c:274 #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106 #6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0 #7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7f4b54faf700 (LWP 6634)): #0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350 #2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a8228) at beam/erl_threads.h:697 #3 async_get (arg=0x7f4b557a8228) at beam/erl_async.c:207 #4 async_main (arg=0x7f4b557a8228) at beam/erl_async.c:274 #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106 #6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0 #7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7f4b54ee3700 (LWP 6640)): #0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350 #2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a84f8) at beam/erl_threads.h:697 #3 async_get (arg=0x7f4b557a84f8) at beam/erl_async.c:207 #4 async_main (arg=0x7f4b557a84f8) at beam/erl_async.c:274 #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106 #6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0 #7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7f4b56239700 (LWP 6629)): #0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 -- Type <return> to continue, or q <return> to quit -- #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350 #2 0x0000000000462ad9 in erts_cnd_wait (unused=<value optimized out>) at beam/erl_threads.h:697 #3 erts_smp_cnd_wait (unused=<value optimized out>) at beam/erl_smp.h:472 #4 sys_msg_dispatcher_func (unused=<value optimized out>) at beam/erl_trace.c:3238 #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5ff0) at pthread/ethread.c:106 #6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0 #7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f4b54f8d700 (LWP 6635)): #0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350 #2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a82a0) at beam/erl_threads.h:697 #3 async_get (arg=0x7f4b557a82a0) at beam/erl_async.c:207 #4 async_main (arg=0x7f4b557a82a0) at beam/erl_async.c:274 #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106 #6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0 #7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f4b0062c700 (LWP 6660)): #0 0x000000344e6ab15d in nanosleep () from /lib64/libc.so.6 #1 0x000000344e6df124 in usleep () from /lib64/libc.so.6 #2 0x00007f4b00631397 in terminatorLoop (args=<value optimized out>) at mapreduce_nif.cc:445 #3 0x0000000000584d89 in thr_wrapper (vtwd=0x7f4b54dd0ae0) at pthread/ethread.c:106 #4 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0 #5 0x000000344e6e5ccd in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f4b54f05700 (LWP 6639)): #0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350 #2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a8480) at beam/erl_threads.h:697 #3 async_get (arg=0x7f4b557a8480) at beam/erl_async.c:207 #4 async_main (arg=0x7f4b557a8480) at beam/erl_async.c:274 #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106 #6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0 #7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f4b55015700 (LWP 6631)): #0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350 #2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a80c0) at beam/erl_threads.h:697 #3 async_get (arg=0x7f4b557a80c0) at beam/erl_async.c:207 #4 async_main (arg=0x7f4b557a80c0) at beam/erl_async.c:274 #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106 #6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0 #7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f4af3171700 (LWP 7145)): #0 0x000000344e6ab15d in nanosleep () from /lib64/libc.so.6 #1 0x000000344e6df124 in usleep () from /lib64/libc.so.6 #2 0x00007f4b50ae7848 in v8::internal::ContextSwitcher::Run (this=0x7f4b0414ace0) at src/v8threads.cc:457 #3 0x00007f4b50a67580 in v8::internal::ThreadEntry (arg=0x7f4b0414ace0) at src/platform-linux.cc:745 #4 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0 #5 0x000000344e6e5ccd in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f4b56f4d700 (LWP 6627)): #0 0x000000344ee0e4ed in read () from /lib64/libpthread.so.0 #1 0x000000000053810e in signal_dispatcher_thread_func (unused=<value optimized out>) at sys/unix/sys.c:2906 #2 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5ee0) at pthread/ethread.c:106 #3 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0 #4 0x000000344e6e5ccd in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f4b54ec1700 (LWP 6641)): #0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 -- Type <return> to continue, or q <return> to quit -- #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350 #2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a8570) at beam/erl_threads.h:697 #3 async_get (arg=0x7f4b557a8570) at beam/erl_async.c:207 #4 async_main (arg=0x7f4b557a8570) at beam/erl_async.c:274 #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106 #6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0 #7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f4b54f49700 (LWP 6637)): #0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350 #2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a8390) at beam/erl_threads.h:697 #3 async_get (arg=0x7f4b557a8390) at beam/erl_async.c:207 #4 async_main (arg=0x7f4b557a8390) at beam/erl_async.c:274 #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106 #6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0 #7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f4b57279700 (LWP 6626)): #0 0x000000344e6de8b3 in select () from /lib64/libc.so.6 #1 0x0000000000537ba0 in erts_sys_main_thread () at sys/unix/sys.c:3049 #2 0x0000000000444761 in erl_start (argc=54, argv=<value optimized out>) at beam/erl_init.c:1441 #3 0x0000000000426219 in main (argc=<value optimized out>, argv=<value optimized out>) at sys/unix/erl_main.c:29 Thread 7 (Thread 0x7f4b54f6b700 (LWP 6636)): #0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350 #2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a8318) at beam/erl_threads.h:697 #3 async_get (arg=0x7f4b557a8318) at beam/erl_async.c:207 #4 async_main (arg=0x7f4b557a8318) at beam/erl_async.c:274 #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106 #6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0 #7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f4b54fd1700 (LWP 6633)): #0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350 #2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a81b0) at beam/erl_threads.h:697 #3 async_get (arg=0x7f4b557a81b0) at beam/erl_async.c:207 #4 async_main (arg=0x7f4b557a81b0) at beam/erl_async.c:274 #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106 #6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0 #7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f4b54e17700 (LWP 6646)): #0 0x000000344ee0f03d in waitpid () from /lib64/libpthread.so.0 #1 0x000000000053906f in child_waiter (unused=<value optimized out>) at sys/unix/sys.c:2821 #2 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5fa0) at pthread/ethread.c:106 #3 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0 #4 0x000000344e6e5ccd in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f4b55037700 (LWP 6630)): #0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350 #2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a8048) at beam/erl_threads.h:697 #3 async_get (arg=0x7f4b557a8048) at beam/erl_async.c:207 #4 async_main (arg=0x7f4b557a8048) at beam/erl_async.c:274 #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106 #6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0 #7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f4b54e5b700 (LWP 6644)): #0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 -- Type <return> to continue, or q <return> to quit -- #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350 #2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a86d8) at beam/erl_threads.h:697 #3 async_get (arg=0x7f4b557a86d8) at beam/erl_async.c:207 #4 async_main (arg=0x7f4b557a86d8) at beam/erl_async.c:274 #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106 #6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0 #7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f4b54e39700 (LWP 6645)): #0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350 #2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a8750) at beam/erl_threads.h:697 #3 async_get (arg=0x7f4b557a8750) at beam/erl_async.c:207 #4 async_main (arg=0x7f4b557a8750) at beam/erl_async.c:274 #5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106 #6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0 #7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f4b54e7d700 (LWP 6643)): #0 right_rotate (allctr=<value optimized out>, del=<value optimized out>, flags=<value optimized out>) at beam/erl_bestfit_alloc.c:252 #1 tree_delete (allctr=<value optimized out>, del=<value optimized out>, flags=<value optimized out>) at beam/erl_bestfit_alloc.c:496 #2 0x00000000004370db in mbc_free (allctr=0xa7c2c0, p=<value optimized out>) at beam/erl_alloc_util.c:883 #3 0x000000000043833f in do_erts_alcu_free (type=<value optimized out>, unused=<value optimized out>, p=0xa87a38) at beam/erl_alloc_util.c:2898 #4 erts_alcu_free_thr_pref (type=<value optimized out>, unused=<value optimized out>, p=0xa87a38) at beam/erl_alloc_util.c:2949 #5 0x000000000054cac0 in file_async_ready (e=0xa84238, data=0xa87a38) at drivers/common/efile_drv.c:2091 #6 0x0000000000480094 in async_ready (p=0x7f4b553521e0, data=<value optimized out>) at beam/io.c:2445 #7 0x00000000004e906f in async_main (arg=0x7f4b557a8660) at beam/erl_async.c:296 #8 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106 #9 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0 #10 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
        Show
        thuan Thuan Nguyen added a comment - Raw erlang core file is here https://s3.amazonaws.com/packages.couchbase/core-dump/orange/201209/core.beam.smp.6626.gz
        Hide
        karan Karan Kumar (Inactive) added a comment -

        Another crash.

        We are seeing it most deterministically.
        https://friendpaste.com/55MrWtpTIufnEMRrxXbLsI

        Show
        karan Karan Kumar (Inactive) added a comment - Another crash. We are seeing it most deterministically. https://friendpaste.com/55MrWtpTIufnEMRrxXbLsI
        damien damien made changes -
        Assignee Damien Katz [ damien ] Aaron Miller [ aaron ]
        Hide
        thuan Thuan Nguyen added a comment -

        Got another erlang segfault during swap rebalance in new system test setup with all SSD drive and consistent view enable
        Erlang crashed on node added to cluster

        Link to stack trace of erlang crash node https://friendpaste.com/58uGMaPS8xo2tcLsDSTTTM

        Show
        thuan Thuan Nguyen added a comment - Got another erlang segfault during swap rebalance in new system test setup with all SSD drive and consistent view enable Erlang crashed on node added to cluster Link to stack trace of erlang crash node https://friendpaste.com/58uGMaPS8xo2tcLsDSTTTM
        aaron Aaron Miller (Inactive) made changes -
        Status Open [ 1 ] In Progress [ 3 ]
        damien damien made changes -
        Assignee Aaron Miller [ aaron ] Damien Katz [ damien ]
        Hide
        thuan Thuan Nguyen added a comment -

        Stack trace erlang crash on build 2.0.0-1832 on system test

        https://friendpaste.com/59jNoNCWLSrPOyTCQK76Rq

        Show
        thuan Thuan Nguyen added a comment - Stack trace erlang crash on build 2.0.0-1832 on system test https://friendpaste.com/59jNoNCWLSrPOyTCQK76Rq
        Hide
        FilipeManana Filipe Manana (Inactive) added a comment -

        Tony, do you think you can save all the files (database, indexes, etc) from the moment the crash happens?
        I think it's more helpful rather than pasting a stack trace everytime it happens.

        Show
        FilipeManana Filipe Manana (Inactive) added a comment - Tony, do you think you can save all the files (database, indexes, etc) from the moment the crash happens? I think it's more helpful rather than pasting a stack trace everytime it happens.
        Hide
        FilipeManana Filipe Manana (Inactive) added a comment -

        Tony, does this always happen as well on the same machine (physical or virtual) ?

        Show
        FilipeManana Filipe Manana (Inactive) added a comment - Tony, does this always happen as well on the same machine (physical or virtual) ?
        Hide
        thuan Thuan Nguyen added a comment -

        Crash did not happen on the same vm. It happened not only in our vms but also in ec2 too as I did test last time

        Show
        thuan Thuan Nguyen added a comment - Crash did not happen on the same vm. It happened not only in our vms but also in ec2 too as I did test last time
        Hide
        thuan Thuan Nguyen added a comment - - edited

        Hit erlang crash on toybuild 10112 on system test.
        Link to stack trace on node 39 https://friendpaste.com/59jNoNCWLSrPOyTCQKWoCO
        Core dump is on /data on node 39.

        Cluster information:

        • 8 centos 6.2 64bit server with 4 cores CPU
        • Each server has 32 GB RAM and 400 GB SSD disk.
        • 24.8 GB RAM for couchbase server at each node
        • SSD disk format ext4 on /data
        • Each server has its own SSD drive, no disk sharing with other server.
        • Create cluster with 6 nodes installed couchbase server 2.0.0-10112 (toy build)
        • Cluster has 2 buckets, default (12GB) and saslbucket (12GB).
        • Each bucket has one doc and 2 views for each doc (default d1 and saslbucket d11)
        • Disable consistent view on cluster.

        10.6.2.37
        10.6.2.38
        10.6.2.39
        10.6.2.40
        10.6.2.44
        10.6.2.45
        10.6.2.42
        10.6.2.43

        • Load 14 million items to both bucket. Each key has size from 512 bytes to 1024 bytes
        • Queries all 4 views from 2 docs
        • Mutate 14 million items with key size from 1500 to 1024 bytes
        Show
        thuan Thuan Nguyen added a comment - - edited Hit erlang crash on toybuild 10112 on system test. Link to stack trace on node 39 https://friendpaste.com/59jNoNCWLSrPOyTCQKWoCO Core dump is on /data on node 39. Cluster information: 8 centos 6.2 64bit server with 4 cores CPU Each server has 32 GB RAM and 400 GB SSD disk. 24.8 GB RAM for couchbase server at each node SSD disk format ext4 on /data Each server has its own SSD drive, no disk sharing with other server. Create cluster with 6 nodes installed couchbase server 2.0.0-10112 (toy build) Cluster has 2 buckets, default (12GB) and saslbucket (12GB). Each bucket has one doc and 2 views for each doc (default d1 and saslbucket d11) Disable consistent view on cluster. 10.6.2.37 10.6.2.38 10.6.2.39 10.6.2.40 10.6.2.44 10.6.2.45 10.6.2.42 10.6.2.43 Load 14 million items to both bucket. Each key has size from 512 bytes to 1024 bytes Queries all 4 views from 2 docs Mutate 14 million items with key size from 1500 to 1024 bytes Do swap rebalance 2 nodes at the same time. Cluster is left as in crashed state to investigate Link to collect info of all nodes https://s3.amazonaws.com/packages.couchbase/collect_info/orange/2_0_0/201210/8nodes-col-toybuild-10112-beam-segfault-20121012-003212.tgz
        Hide
        damien damien added a comment -

        Right now we think this might be related to another bug spotted on Erlang VMs. When started with async threads and using "raw" file descriptors, if the process that opened the file is shutdown or crashes abnormally, the the file descriptor is leaked.

        one possible fix is to change the erlang startup parameters to turn off async file io, so this:
        > erl +A 16 +sbt u +P 327680 +K true
        becomes:
        > erl +sbt u +P 327680 +K true

        Show
        damien damien added a comment - Right now we think this might be related to another bug spotted on Erlang VMs. When started with async threads and using "raw" file descriptors, if the process that opened the file is shutdown or crashes abnormally, the the file descriptor is leaked. one possible fix is to change the erlang startup parameters to turn off async file io, so this: > erl +A 16 +sbt u +P 327680 +K true becomes: > erl +sbt u +P 327680 +K true
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        Just keep in mind that without async io we'll have massive timeouts all over the place. We specifically had exactly this problem in early days of 1.6.0.

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - Just keep in mind that without async io we'll have massive timeouts all over the place. We specifically had exactly this problem in early days of 1.6.0.
        Hide
        farshid Farshid Ghods (Inactive) added a comment -

        according to karan he has not seen any crash or rebalance timeouts on cluster where there are lot of rebalancing
        1- 2.0 cluster doing views , 2 buckets , 30M items , 2 ddocs , 2 views , 8 nodes under 30k ops/sec and 300 queries per second
        2- 20+ node cluster running 1.8.x key value use cases

        performance team is also running more performance tests to verify that +s12 works ( without +a) option

        we have not yet run xdcr system tests with these settings.

        Show
        farshid Farshid Ghods (Inactive) added a comment - according to karan he has not seen any crash or rebalance timeouts on cluster where there are lot of rebalancing 1- 2.0 cluster doing views , 2 buckets , 30M items , 2 ddocs , 2 views , 8 nodes under 30k ops/sec and 300 queries per second 2- 20+ node cluster running 1.8.x key value use cases performance team is also running more performance tests to verify that +s12 works ( without +a) option we have not yet run xdcr system tests with these settings.
        Hide
        ketaki Ketaki Gangal added a comment -

        Tested xdcr-functional tests with the setting +S 120:120, it works fine.

        Show
        ketaki Ketaki Gangal added a comment - Tested xdcr-functional tests with the setting +S 120:120, it works fine.
        Hide
        steve Steve Yen added a comment -

        Notes from mtg with damien, alk, farshid, sriram, sharon, aaron, yaseen, aaron

        priority: key-value use-case cannot regress from 1.8.1

        plan A - more testing with no-async (farshid).

        • +S 120:120 (not 12:12)
        • QE run tests on real disk & durable configuration setting.
        • look for timeouts in the U/I.
        • windows.
        • use physical disk boxes from perf team.

        plan B - track down root cause.

        • QE will run system tests against toy-build from siri, on VMs ok.
        • instrument memory calls, look for culprit - toy build (siri).
        • idea - hammer NIF erlang unit tests repeatedly overnight (alk).
        • idea - try "A1" (try this if plan A doesn't work).
          + frees up scheduler threads.
          + and single async-i/o thread might mitigate crashes.
        • this will serialize I/O even if we have multiple disks.

        plan C - separate processes (if plan A doesn't work, and alk's team has time).
        + should fix the problem.

        • costly.
        • some tight coupling between some modules.

        plan D - run async for key-value use case, sync for view use case.

        -----------
        by the way, 1.8.1 used async threads.

        async threads mode can leak file descriptors.

        • this was some different issue.
        Show
        steve Steve Yen added a comment - Notes from mtg with damien, alk, farshid, sriram, sharon, aaron, yaseen, aaron priority: key-value use-case cannot regress from 1.8.1 plan A - more testing with no-async (farshid). +S 120:120 (not 12:12) QE run tests on real disk & durable configuration setting. look for timeouts in the U/I. windows. use physical disk boxes from perf team. plan B - track down root cause. QE will run system tests against toy-build from siri, on VMs ok. instrument memory calls, look for culprit - toy build (siri). idea - hammer NIF erlang unit tests repeatedly overnight (alk). idea - try "A1" (try this if plan A doesn't work). + frees up scheduler threads. + and single async-i/o thread might mitigate crashes. this will serialize I/O even if we have multiple disks. plan C - separate processes (if plan A doesn't work, and alk's team has time). + should fix the problem. costly. some tight coupling between some modules. plan D - run async for key-value use case, sync for view use case. ----------- by the way, 1.8.1 used async threads. async threads mode can leak file descriptors. this was some different issue.
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        Just verified perf team's RHEL 5.8 boxes. Indeed RHEL 5.8 even on ext3 has barriers ON by default. That's good for me.

        I tested with my tool and I'm seeing typical performance from 7.2k RPMs disk.

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - Just verified perf team's RHEL 5.8 boxes. Indeed RHEL 5.8 even on ext3 has barriers ON by default. That's good for me. I tested with my tool and I'm seeing typical performance from 7.2k RPMs disk.
        Hide
        ronnie Ronnie Sun (Inactive) added a comment -

        btw, the test we were running used +S 16:16 instead of 12:12

        Show
        ronnie Ronnie Sun (Inactive) added a comment - btw, the test we were running used +S 16:16 instead of 12:12
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        Tested on /data2 partition that's on lvm. And found that barriers actually don't work.

        So system tests physical box is NOT ok. Looks like older (or maybe even modern) LVM does not provide durable disk configuration.

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - Tested on /data2 partition that's on lvm. And found that barriers actually don't work. So system tests physical box is NOT ok. Looks like older (or maybe even modern) LVM does not provide durable disk configuration.
        Hide
        damien damien added a comment -

        Mounting evidence suggests this bug is caused by the +A erlang startup settings, which turns on async IO for port drivers by using a pool of threads to perform the IO. We still don't understand the root cause, but it appears there is a race condition/cache coherency problem with how port drivers are freed in the VM.

        Reassigning to Alk, as he will check in the change to disable the async threads and bump up the # of schedulers to mitigate problems with timeouts due to blocking IO.

        Show
        damien damien added a comment - Mounting evidence suggests this bug is caused by the +A erlang startup settings, which turns on async IO for port drivers by using a pool of threads to perform the IO. We still don't understand the root cause, but it appears there is a race condition/cache coherency problem with how port drivers are freed in the VM. Reassigning to Alk, as he will check in the change to disable the async threads and bump up the # of schedulers to mitigate problems with timeouts due to blocking IO.
        damien damien made changes -
        Assignee Damien Katz [ damien ] Aleksey Kondratenko [ alkondratenko ]
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        We merged "async off" commit.

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - We merged "async off" commit.
        alkondratenko Aleksey Kondratenko (Inactive) made changes -
        Assignee Aleksey Kondratenko [ alkondratenko ] Damien Katz [ damien ]
        damien damien made changes -
        Status In Progress [ 3 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        kzeller kzeller added a comment -

        Added to RN : Couchbase Server had intermittently crashed
        during rebalance due to Erlang virtual machine
        issues; we now
        disable asynchronous threads and perform garbage collection
        more often to avoid timeouts and process crashes.

        Show
        kzeller kzeller added a comment - Added to RN : Couchbase Server had intermittently crashed during rebalance due to Erlang virtual machine issues; we now disable asynchronous threads and perform garbage collection more often to avoid timeouts and process crashes.
        FilipeManana Filipe Manana (Inactive) made changes -
        Component/s 3rd-party [ 10219 ]
        Hide
        thuan Thuan Nguyen added a comment -

        Integrated in github-couchdb-preview #544 (See http://qa.hq.northscale.net/job/github-couchdb-preview/544/)
        MB-6638: use fork of file_sorter that doesn't use compression (Revision 6bbe1cf89b2f6b5c9cf098b81c5ea60d339f8f0a)

        Result = SUCCESS
        Aliaksey Kandratsenka :
        Files :

        • src/couch_set_view/src/couch_set_view_updater.erl
        • src/couchdb/couch_db_updater.erl
        • src/couchdb/Makefile.am
        • Makefile.am
        • src/couchdb/file_sorter_2.erl
        Show
        thuan Thuan Nguyen added a comment - Integrated in github-couchdb-preview #544 (See http://qa.hq.northscale.net/job/github-couchdb-preview/544/ ) MB-6638 : use fork of file_sorter that doesn't use compression (Revision 6bbe1cf89b2f6b5c9cf098b81c5ea60d339f8f0a) Result = SUCCESS Aliaksey Kandratsenka : Files : src/couch_set_view/src/couch_set_view_updater.erl src/couchdb/couch_db_updater.erl src/couchdb/Makefile.am Makefile.am src/couchdb/file_sorter_2.erl
        farshid Farshid Ghods (Inactive) made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            damien damien
            Reporter:
            thuan Thuan Nguyen
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes