forestdb rollback panic

System test on forestdb fails (build 2584)

1) Forestdb commit at seqno 1649738

2016-05-19T13:19:29.322-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 1061387876680995796 Creating New Snapshot SnapshotInfo: seqnos: 1649738, 1649738, 1 committed:true
2016-05-19T13:19:29.322-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 1061387876680995796 PartitionId: 0 SliceId: 0 Crc64: 11244626529134686691 (SnapshotInfo: seqnos: 1649738, 1649738, 1 committed:true) SnapCreateDur 19.438599ms SnapOpenDur 90.886µs

2) Forestdb commit at seqno 1658760 (last commit before crash)

2016-05-19T13:24:57.893-07:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 1061387876680995796 FlushTime 41.993µs CommitTime 40.454535ms TotalFlushTime 2m34.237364189s TotalCommitTime 30.812898727s
2016-05-19T13:24:57.893-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 1061387876680995796 Creating New Snapshot SnapshotInfo: seqnos: 1658760, 1658760, 1 committed:true
2016-05-19T13:24:57.893-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 1061387876680995796 PartitionId: 0 SliceId: 0 Crc64: 7600448576807373898 (SnapshotInfo: seqnos: 1658760, 1658760, 1 committed:true) SnapCreateDur 52.327652ms SnapOpenDur 151.876µs

2016-05-19T13:24:58.185-07:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 1061387876680995796 Creating New Snapshot SnapshotInfo: seqnos: 1666396, 1666396, 0 committed:false
2016-05-19T13:24:58.188-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 1061387876680995796 PartitionId: 0 SliceId: 0 Crc64: 7600448576807373898 (SnapshotInfo: seqnos: 1666396, 1666396, 0 committed:false) SnapCreateDur 50.764µs SnapOpenDur 3.043584ms

3) Indexer tries to rollback to the last (oldest) persistent snapshot.

2016-05-19T13:24:58.468-07:00 [Info] SnapshotContainer::GetOlderThanTS Snapshot List: [SnapshotInfo: seqnos: 1658760, 1658760, 1 committed:true SnapshotInfo: seqnos: 1655452, 1655452, 1 committed:true SnapshotInfo: seqnos: 1654572, 1654572, 1 committed:true SnapshotInfo: seqnos: 1652701, 1652701, 1 committed:true SnapshotInfo: seqnos: 1649738, 1649738, 1 committed:true]

4) Rollback panic

2016-05-19T13:24:58.765-07:00 [Error] ForestDBSlice::Rollback
SliceId 0 IndexInstId 1061387876680995796. Error Rollback Meta Index to Snapshot SnapshotInfo: seqnos: 1649738, 1649738, 1 committed:true. Error FDB_RESULT_NO_DB_INSTANCE
2016-05-19T13:24:58.765-07:00 [Fatal] Indexer::processRollback Error during Rollback FDB_RESULT_NO_DB_INSTANCE
panic: FDB_RESULT_NO_DB_INSTANCE