Page 1 of 3

Records saving

Posted: 18 Aug 2011, 19:33
by lgromanowski
Hi,
I'm considering to take Record saving task.
I have following (rough) plan, but is there anything else which should I take for consideration?
  1. Rename components/esm/load*.[hpp|cpp] into components/esm/classname.[hpp|cpp] (it's not mentioned in coding policies, but when I add store() method then file name become misleading)
  2. Rename esm_reader* to esmreader* (according to Naming conventions)
  3. Write esmwriter class (based on esm_reader)
  4. For each load*.[hpp|cpp] write a store() method
  5. Prepare a tests (maybe we could use CppUnit or Google C++ testing framework?)

Re: Records saving

Posted: 18 Aug 2011, 19:57
by Zini
Okay on #1, #2 and #4.

#3 looks like overkill, since we don't write esms from the data structures we have in components. We need to write in two cases:

1. Save game (dynamically generated records only): This will probably use a new file format; maybe similar to esm/p. But we haven't designed it yet and we won't write the data from the default record store anyway.

2. Save esm/p from the editor: The editor will use a different kind of data storage (components/esm yes, components/esm_store no).

We may need something like an esmwriter for the testing part, but I suspect that we can get away with writing out each record immediately after we have read it in.

About #5: I am considering revamping the esm tool: Clean up the command line interface and add commands similar in style to git. The current function could be handled by a "dump" command. And the testing by a "test" command, which would read in an esm, write it to a separate location (on disc or in memeory), read it in again and compare this with what was read originally.
This approach might be more useful in the long run. But if you are more comfortable with using a testing framework, that would be okay too.

P.S.: I think Corristo has some code for handling some terrain-related record types that are not available in master yet. Maybe you should pull them in (cherry pick?) .

Edit:

P.P.S.: Also, I think jhooks has a fix for one animation related record in his branch. Cherry pick too?

Re: Records saving

Posted: 18 Aug 2011, 20:20
by lgromanowski
Zini wrote:Okay on #1, #2 and #4.

#3 looks like overkill, since we don't write esms from the data structures we have in components. We need to write in two cases:

1. Save game (dynamically generated records only): This will probably use a new file format; maybe similar to esm/p. But we haven't designed it yet and we won't write the data from the default record store anyway.

2. Save esm/p from the editor: The editor will use a different kind of data storage (components/esm yes, components/esm_store no).

We may need something like an esmwriter for the testing part, but I suspect that we can get away with writing out each record immediately after we have read it in.
OK.
Zini wrote: About #5: I am considering revamping the esm tool: Clean up the command line interface and add commands similar in style to git. The current function could be handled by a "dump" command. And the testing by a "test" command, which would read in an esm, write it to a separate location (on disc or in memeory), read it in again and compare this with what was read originally.
This approach might be more useful in the long run. But if you are more comfortable with using a testing framework, that would be okay too.
OK, I will do both - revamp the esm tool and prepare a unit tests.
Zini wrote: P.S.: I think Corristo has some code for handling some terrain-related record types that are not available in master yet. Maybe you should pull them in (cherry pick?) .
OK, I will cherypick them.

Re: Records saving

Posted: 18 Aug 2011, 20:34
by Zini
I hope you saw my PPS in the edit.

Re: Records saving

Posted: 18 Aug 2011, 21:05
by jhooks1
Zini wrote:
P.P.S.: Also, I think jhooks has a fix for one animation related record in his branch. Cherry pick too?
Yes, check the commit "Fixed part list adding"

Re: Records saving

Posted: 18 Aug 2011, 21:17
by lgromanowski
OK, thanks, I will check jhooks commit too.

Re: Records saving

Posted: 24 Sep 2011, 09:12
by Zini
@lgro: Since you were planning to revamp the esmtool anyway, could you have a look at this bug I found:
esmtool: /home/marc/OpenMW/openmw/components/to_utf8/to_utf8.cpp:152: std::string ToUTF8::getUtf8(ToUTF8::FromType): Assertion `0' failed.
Happens when running the esmtool with a non-English Morrowind.esm. I assume the encoding fixes in OpenMW broke something in the tool (probably just requires an option to specify encoding or something like that).

Re: Records saving

Posted: 01 Oct 2011, 12:39
by lgromanowski
Zini wrote:@lgro: Since you were planning to revamp the esmtool anyway, could you have a look at this bug I found:
esmtool: /home/marc/OpenMW/openmw/components/to_utf8/to_utf8.cpp:152: std::string ToUTF8::getUtf8(ToUTF8::FromType): Assertion `0' failed.
Happens when running the esmtool with a non-English Morrowind.esm. I assume the encoding fixes in OpenMW broke something in the tool (probably just requires an option to specify encoding or something like that).
Sorry for late reply. Yes I will look into it.

Re: Records saving

Posted: 01 Mar 2012, 09:08
by werdanith
Resurrecting this thread since I was planning to use the esmtool, and then found out that it doesn't print the records of weapons. First though a couple of issues I wanted to tackle.

issue #1:
Zini wrote:@lgro: Since you were planning to revamp the esmtool anyway, could you have a look at this bug I found:
esmtool: /home/marc/OpenMW/openmw/components/to_utf8/to_utf8.cpp:152: std::string ToUTF8::getUtf8(ToUTF8::FromType): Assertion `0' failed.
Happens when running the esmtool with a non-English Morrowind.esm. I assume the encoding fixes in OpenMW broke something in the tool (probably just requires an option to specify encoding or something like that).
I had that issue too, and fixed it up temporarily to get the tool working, I will have to look into a more permanent solution now.


issue #2:
When you compile the game in release mode it throws this warning a million times (due to the optimizer):

Code: Select all

components/esm/esm_reader.hpp: In member function ‘float ESM::ESMReader::getFVer()’:
components/esm/esm_reader.hpp:156:58: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
which comes from this line:

Code: Select all

float getFVer() { return *((float*)&mCtx.header.version); }
It turns out that we have a struct ESM_Context whose Head sub-member has a integer member named version, which is actually storing a float and we are using this hacky cast to convert it back to float when we request it.

Not sure how to proceed with this one.
There are only two possible values for the version:

Code: Select all

enum Version
  {
    VER_12 = 0x3f99999a,
    VER_13 = 0x3fa66666
  };
so one solution would be to convert the function into something like this:

Code: Select all

float getFVer() {  if(mCtx.header.version == VER_12) return 1.2; else return 1.3;  }
ideas?

Re: Records saving

Posted: 01 Mar 2012, 09:19
by werdanith
In retrospect, starting a new thread and adding a link for the quote I wanted was probably a better idea.