ns_server takes too long to configure indexes and attempts to use them before configuring them

On latest master branch of everything (sunday August 19), start a 4 nodes cluster and run the following testrunner test:

$ ./testrunner -i b/resources/dev-4-nodes.ini -t viewtests.ViewBasicTests.test_load_data_get_view_x_mins_y_design_docs,num-design-docs=20,load-time=10,run-view-time=1,num-tries=100

For a long time, view queries get a view_undefined error for many ddocs, and internally ns_server attempts to update the index partition states before configuring the indexes.
Example for one run, log debug.1 is from node 0, design document _design/dev_test_view_on_10000_docs-571a9ab. Analysis follows:

Design document _design/dev_test_view_on_10000_docs-571a9ab

Created at (line 14413):

[ns_server:debug,2012-08-19T18:33:28.381,n_0@192.168.1.87:'capi_ddoc_replication_srv-default':cb_generic_replication_srv:handle_call:92]Writing interactively saved ddoc {doc,<<"_design/dev_test_view_on_10000_docs-571a9ab">>,

{0,<<>>}

,
<<"{\"views\":{\"dev_test_view_on_10000_docs-571a9ab\":{\"map\":\"function (doc) {if(doc.name.indexOf(\\\"dev_test_view_on_10000_docs-571a9ab\\\") != -1) { emit(doc.name, doc);}}\"}}}">>,
0,false,[]}

and at line 14425:

[views:debug,2012-08-19T18:33:29.041,n_0@192.168.1.87:<0.688.0>:capi_set_view_manager:master_db_watcher:403]Got change in `_design/dev_test_view_on_10000_docs-571a9ab` design document. Initiating set view update

Then group starts at line 14468:

[couchdb:info,2012-08-19T18:33:29.234,n_0@192.168.1.87:<0.9246.0>:couch_log:info:39]Started undefined main set view group `default`, group `_design/dev_test_view_on_10000_docs-571a9ab`, signature `c1f7b3ace0cf88e8ad256985615470dd', view count: 1

However before it gets configured, capi_set_view_manager attempts to use it several times, and therefore getting view_undefined errors:

example 1 (line 14914):

Calling couch_set_view:set_partition_states([<<"default">>,
<<"_design/dev_test_view_on_10000_docs-571a9ab">>,
[0,1,2,3,4,5,6,7,8,9,10,11,12,13,
14,15],
[],
[16,17,18,19,20,21,22,23,24,25,
26,27,28,29,30,31,32,33,34,35,
36,37,38,39,40,41,42,43,44,45,
46,47,48,49,50,51,52,53,54,55,
56,57,58,59,60,61,62,63]])
[views:info,2012-08-19T18:33:29.949,n_0@192.168.1.87:'capi_ddoc_replication_srv-default':capi_set_view_manager:apply_index_states:334]
couch_set_view:set_partition_states([<<"default">>,
<<"_design/dev_test_view_on_10000_docs-571a9ab">>,
[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15],
[],
[16,17,18,19,20,21,22,23,24,25,26,27,28,
29,30,31,32,33,34,35,36,37,38,39,40,41,
42,43,44,45,46,47,48,49,50,51,52,53,54,
55,56,57,58,59,60,61,62,63]]) raised throw:

{error, view_undefined}

example 2 (line 15502):

[views:info,2012-08-19T18:33:31.392,n_0@192.168.1.87:'capi_ddoc_replication_srv-default':capi_set_view_manager:apply_index_states:333]
Calling couch_set_view:set_partition_states([<<"default">>,
<<"_design/dev_test_view_on_10000_docs-571a9ab">>,
[0,1,2,3,4,5,6,7,8,9,10,11,12,13,
14,15],
[],
[16,17,18,19,20,21,22,23,24,25,
26,27,28,29,30,31,32,33,34,35,
36,37,38,39,40,41,42,43,44,45,
46,47,48,49,50,51,52,53,54,55,
56,57,58,59,60,61,62,63]])
[views:info,2012-08-19T18:33:31.420,n_0@192.168.1.87:'capi_ddoc_replication_srv-default':capi_set_view_manager:apply_index_states:334]
couch_set_view:set_partition_states([<<"default">>,
<<"_design/dev_test_view_on_10000_docs-571a9ab">>,
[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15],
[],
[16,17,18,19,20,21,22,23,24,25,26,27,28,
29,30,31,32,33,34,35,36,37,38,39,40,41,
42,43,44,45,46,47,48,49,50,51,52,53,54,
55,56,57,58,59,60,61,62,63]]) raised throw:{error, view_undefined}

example 3 (line 16205):

[views:info,2012-08-19T18:33:33.064,n_0@192.168.1.87:'capi_ddoc_replication_srv-default':capi_set_view_manager:apply_index_states:333]
Calling couch_set_view:set_partition_states([<<"default">>,
<<"_design/dev_test_view_on_10000_docs-571a9ab">>,
[0,1,2,3,4,5,6,7,8,9,10,11,12,13,
14,15],
[],
[16,17,18,19,20,21,22,23,24,25,
26,27,28,29,30,31,32,33,34,35,
36,37,38,39,40,41,42,43,44,45,
46,47,48,49,50,51,52,53,54,55,
56,57,58,59,60,61,62,63]])
[views:info,2012-08-19T18:33:33.066,n_0@192.168.1.87:'capi_ddoc_replication_srv-default':capi_set_view_manager:apply_index_states:334]
couch_set_view:set_partition_states([<<"default">>,
<<"_design/dev_test_view_on_10000_docs-571a9ab">>,
[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15],
[],
[16,17,18,19,20,21,22,23,24,25,26,27,28,
29,30,31,32,33,34,35,36,37,38,39,40,41,
42,43,44,45,46,47,48,49,50,51,52,53,54,
55,56,57,58,59,60,61,62,63]]) raised throw:

{error, view_undefined}

example 4 (line 16930):

[views:info,2012-08-19T18:33:34.914,n_0@192.168.1.87:'capi_ddoc_replication_srv-default':capi_set_view_manager:apply_index_states:333]
Calling couch_set_view:set_partition_states([<<"default">>,
<<"_design/dev_test_view_on_10000_docs-571a9ab">>,
[0,1,2,3,4,5,6,7,8,9,10,11,12,13,
14,15],
[],
[16,17,18,19,20,21,22,23,24,25,
26,27,28,29,30,31,32,33,34,35,
36,37,38,39,40,41,42,43,44,45,
46,47,48,49,50,51,52,53,54,55,
56,57,58,59,60,61,62,63]])
[views:info,2012-08-19T18:33:34.923,n_0@192.168.1.87:'capi_ddoc_replication_srv-default':capi_set_view_manager:apply_index_states:334]
couch_set_view:set_partition_states([<<"default">>,
<<"_design/dev_test_view_on_10000_docs-571a9ab">>,
[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15],
[],
[16,17,18,19,20,21,22,23,24,25,26,27,28,
29,30,31,32,33,34,35,36,37,38,39,40,41,
42,43,44,45,46,47,48,49,50,51,52,53,54,
55,56,57,58,59,60,61,62,63]]) raised throw:{error, view_undefined}

example 5 (line 17713):

[views:info,2012-08-19T18:33:36.896,n_0@192.168.1.87:'capi_ddoc_replication_srv-default':capi_set_view_manager:apply_index_states:333]
Calling couch_set_view:set_partition_states([<<"default">>,
<<"_design/dev_test_view_on_10000_docs-571a9ab">>,
[0,1,2,3,4,5,6,7,8,9,10,11,12,13,
14,15],
[],
[16,17,18,19,20,21,22,23,24,25,
26,27,28,29,30,31,32,33,34,35,
36,37,38,39,40,41,42,43,44,45,
46,47,48,49,50,51,52,53,54,55,
56,57,58,59,60,61,62,63]])
[views:info,2012-08-19T18:33:36.897,n_0@192.168.1.87:'capi_ddoc_replication_srv-default':capi_set_view_manager:apply_index_states:334]
couch_set_view:set_partition_states([<<"default">>,
<<"_design/dev_test_view_on_10000_docs-571a9ab">>,
[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15],
[],
[16,17,18,19,20,21,22,23,24,25,26,27,28,
29,30,31,32,33,34,35,36,37,38,39,40,41,
42,43,44,45,46,47,48,49,50,51,52,53,54,
55,56,57,58,59,60,61,62,63]]) raised throw:

{error, view_undefined}


example 6 (line 18720):

[views:info,2012-08-19T18:33:47.298,n_0@192.168.1.87:'capi_ddoc_replication_srv-default':capi_set_view_manager:apply_index_states:333]
Calling couch_set_view:set_partition_states([<<"default">>,
<<"_design/dev_test_view_on_10000_docs-571a9ab">>,
[0,1,2,3,4,5,6,7,8,9,10,11,12,13,
14,15],
[],
[16,17,18,19,20,21,22,23,24,25,
26,27,28,29,30,31,32,33,34,35,
36,37,38,39,40,41,42,43,44,45,
46,47,48,49,50,51,52,53,54,55,
56,57,58,59,60,61,62,63]])
[views:info,2012-08-19T18:33:47.376,n_0@192.168.1.87:'capi_ddoc_replication_srv-default':capi_set_view_manager:apply_index_states:334]
couch_set_view:set_partition_states([<<"default">>,
<<"_design/dev_test_view_on_10000_docs-571a9ab">>,
[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15],
[],
[16,17,18,19,20,21,22,23,24,25,26,27,28,
29,30,31,32,33,34,35,36,37,38,39,40,41,
42,43,44,45,46,47,48,49,50,51,52,53,54,
55,56,57,58,59,60,61,62,63]]) raised throw:{error, view_undefined}

example 7 (line 20521):

[views:info,2012-08-19T18:34:06.886,n_0@192.168.1.87:'capi_ddoc_replication_srv-default':capi_set_view_manager:apply_index_states:333]
Calling couch_set_view:set_partition_states([<<"default">>,
<<"_design/dev_test_view_on_10000_docs-571a9ab">>,
[0,1,2,3,4,5,6,7,8,9,10,11,12,13,
14,15],
[],
[16,17,18,19,20,21,22,23,24,25,
26,27,28,29,30,31,32,33,34,35,
36,37,38,39,40,41,42,43,44,45,
46,47,48,49,50,51,52,53,54,55,
56,57,58,59,60,61,62,63]])
[views:info,2012-08-19T18:34:06.959,n_0@192.168.1.87:'capi_ddoc_replication_srv-default':capi_set_view_manager:apply_index_states:334]
couch_set_view:set_partition_states([<<"default">>,
<<"_design/dev_test_view_on_10000_docs-571a9ab">>,
[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15],
[],
[16,17,18,19,20,21,22,23,24,25,26,27,28,
29,30,31,32,33,34,35,36,37,38,39,40,41,
42,43,44,45,46,47,48,49,50,51,52,53,54,
55,56,57,58,59,60,61,62,63]]) raised throw:

{error, view_undefined}

example 8 (line 31288):

[views:info,2012-08-19T18:36:58.308,n_0@192.168.1.87:'capi_ddoc_replication_srv-default':capi_set_view_manager:apply_index_states:333]
Calling couch_set_view:set_partition_states([<<"default">>,
<<"_design/dev_test_view_on_10000_docs-571a9ab">>,
[0,1,2,3,4,5,6,7,8,9,10,11,12,13,
14,15],
[],
[16,17,18,19,20,21,22,23,24,25,
26,27,28,29,30,31,32,33,34,35,
36,37,38,39,40,41,42,43,44,45,
46,47,48,49,50,51,52,53,54,55,
56,57,58,59,60,61,62,63]])
[couchdb:info,2012-08-19T18:36:58.351,n_0@192.168.1.87:<0.15476.0>:couch_log:info:39]192.168.1.87 - - GET /default/_design/dev_test_view_on_10000_docs-f21faf9/_view/dev_test_view_on_10000_docs-f21faf9?connection_timeout=60000&full_set=true 200
[views:info,2012-08-19T18:36:58.381,n_0@192.168.1.87:'capi_ddoc_replication_srv-default':capi_set_view_manager:apply_index_states:334]
couch_set_view:set_partition_states([<<"default">>,
<<"_design/dev_test_view_on_10000_docs-571a9ab">>,
[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15],
[],
[16,17,18,19,20,21,22,23,24,25,26,27,28,
29,30,31,32,33,34,35,36,37,38,39,40,41,
42,43,44,45,46,47,48,49,50,51,52,53,54,
55,56,57,58,59,60,61,62,63]]) raised throw:{error, view_undefined}

(etc, many other examples)

And then finally, much later, it configures the view group/ddoc (line 46317):

[views:info,2012-08-19T18:40:35.086,n_0@192.168.1.87:'capi_set_view_manager-default':capi_set_view_manager:define_group:300]
Calling couch_set_view:define_group([<<"default">>,
<<"_design/dev_test_view_on_10000_docs-571a9ab">>,

{set_view_params,64,[],[],true}

])

Seems related to MB-6291, MB-6262 and other issues mentioning view_undefined errors.