Multiple ESMs and ESP support
Posted: 02 Mar 2012, 12:50
This task is scheduled for 0.14.0 and I think lgro wanted to take it.
I would like to do a brain dump before work on it starts.
Basically the task consists of reading repeatedly ESM/ESP files in the predefined order and on each pass:
- adjust modified records in the ESMStore
- add new records to the ESMStore
- delete records from the ESMStore which are flagged as deleted
But there are two special cases that needs to be considered:
1. Cells
Cells are stored in two locations:
1a. the immutable part is stored in the ESMStore
1b. the mutable part (the table of references in the cell) are stored in ESMS::CellStore instances, which are managed by MWWorld::Cells. These are only loaded on demand.
The CellStore uses an optimisation mechanism that allows to quickly load a list of possibly contained IDs (this is required for various scripting functions). This is indicted by the state enum State_Preload. When a cell becomes active, the cell is fully loaded (State_Loaded).
So what needs to be done here is to implement mechanisms to preload and load cells from multiple ESX files.
Preloading should only add IDs, not delete them and not check for duplicates (we are trying to be as fast as possible here).
There might be a problem though. The cell record struct stores an instance of ESM_Context, which indicates a file position for fast access to the cell's record in the ESM file. I haven't looked into the implementation, but if this involves keeping the file open all the time, we have a problem. The number of ESX files can become very large and we probably shouldn't keep that much files open at the same time. So this needs to be looked into.
Also, I think when an ESX file modifies a cell the old reference list is complete discarded. But I could be wrong about it. That is a point that also needs to be checked.
2. Terrain textures
I don't have the details at hand. But one of the record types needs special treatment, because it references another record type, but not from the merged ESX stack, but from the current ESX file. There are probably more details somewhere in the terrain thread.
I suggest the following procedure:
- Implement the general case and special case #1 in a new branch (esx).
- Merge this branch into the terrain branch and implement speical case #2 there.
By doing it this way we make sure we can release the ESX feature, even if the terrain feature is not ready at this point (which mostly depends on how fast the OGRE guys are with getting 1.8 out of RC state).
I would like to do a brain dump before work on it starts.
Basically the task consists of reading repeatedly ESM/ESP files in the predefined order and on each pass:
- adjust modified records in the ESMStore
- add new records to the ESMStore
- delete records from the ESMStore which are flagged as deleted
But there are two special cases that needs to be considered:
1. Cells
Cells are stored in two locations:
1a. the immutable part is stored in the ESMStore
1b. the mutable part (the table of references in the cell) are stored in ESMS::CellStore instances, which are managed by MWWorld::Cells. These are only loaded on demand.
The CellStore uses an optimisation mechanism that allows to quickly load a list of possibly contained IDs (this is required for various scripting functions). This is indicted by the state enum State_Preload. When a cell becomes active, the cell is fully loaded (State_Loaded).
So what needs to be done here is to implement mechanisms to preload and load cells from multiple ESX files.
Preloading should only add IDs, not delete them and not check for duplicates (we are trying to be as fast as possible here).
There might be a problem though. The cell record struct stores an instance of ESM_Context, which indicates a file position for fast access to the cell's record in the ESM file. I haven't looked into the implementation, but if this involves keeping the file open all the time, we have a problem. The number of ESX files can become very large and we probably shouldn't keep that much files open at the same time. So this needs to be looked into.
Also, I think when an ESX file modifies a cell the old reference list is complete discarded. But I could be wrong about it. That is a point that also needs to be checked.
2. Terrain textures
I don't have the details at hand. But one of the record types needs special treatment, because it references another record type, but not from the merged ESX stack, but from the current ESX file. There are probably more details somewhere in the terrain thread.
I suggest the following procedure:
- Implement the general case and special case #1 in a new branch (esx).
- Merge this branch into the terrain branch and implement speical case #2 there.
By doing it this way we make sure we can release the ESX feature, even if the terrain feature is not ready at this point (which mostly depends on how fast the OGRE guys are with getting 1.8 out of RC state).