How to handle type changes?

Discussion and help for Easy Save 3
Manufacture43-Daniel
Posts: 6
Joined: Wed Sep 09, 2020 9:50 am

How to handle type changes?

Post by Manufacture43-Daniel »

Hello,

Recently we made a change to one of our saved data structures: a List<Loot> was changed to Dictionary<string, int>. Loot was a structure with several values half of them were not actually useful in the save... Anyway, this change yielded two issues:

- It raised an exception while loading that was very confusing:

Code: Select all

System.FormatException: Expected '[' or "null", found '{'.
As I could gather from stepping through ES3 code, the loader tries to read the data the way the code is formatted instead of reading data first and then applying it to the loaded fields. If it was done the second way, it would be possible to yield a more sensible error message like "Invalid Data Format" or something.

- I didn't find any way of handling data type changes in ES3. What we did was rename the value to another name so that the loading doesn't fail but that also means that all the data ends up empty in that value... It would be nice to have something equivalent to ISerialization's special constructor that takes in a SerializationInfo and a StreamingContext so that we can manually read data and support old data formats. Is there any way of doing so right now that I may have missed?
User avatar
Joel
Moodkie Staff
Posts: 4826
Joined: Wed Nov 07, 2012 10:32 pm

Re: How to handle type changes?

Post by Joel »

Hi there, and thanks for the feedback.
As I could gather from stepping through ES3 code, the loader tries to read the data the way the code is formatted instead of reading data first and then applying it to the loaded fields. If it was done the second way, it would be possible to yield a more sensible error message like "Invalid Data Format" or something.
We do it this way as it significantly improves performance, otherwise you're essentially doubling your memory overhead. Just to clarify, we follow .NET's conventions when it comes to handling incorrectly formatted data (i.e. by throwing a FormatException).
- I didn't find any way of handling data type changes in ES3. What we did was rename the value to another name so that the loading doesn't fail but that also means that all the data ends up empty in that value... It would be nice to have something equivalent to ISerialization's special constructor that takes in a SerializationInfo and a StreamingContext so that we can manually read data and support old data formats. Is there any way of doing so right now that I may have missed?
We allow you to create ES3Types to do this, which are documented here:
https://docs.moodkie.com/easy-save-3/es ... g-es3types

All the best,
Joel
Joel @ Moodkie Interactive
Purchase Easy Save | Contact | Guides | Docs | Getting started
Manufacture43-Daniel
Posts: 6
Joined: Wed Sep 09, 2020 9:50 am

Re: How to handle type changes?

Post by Manufacture43-Daniel »

I fail to see how implementing an ES3Type here would save us. The code could be something in the lines of reading the variable in its current data format in a try block and then trying to read the old data format in the catch block. But since the stream would have already been consumed by the first try (and actually left in a pretty weird state) I don't think that would work.
Manufacture43-Daniel
Posts: 6
Joined: Wed Sep 09, 2020 9:50 am

Re: How to handle type changes?

Post by Manufacture43-Daniel »

Also, I'm not exactly against the type of exception being thrown, it's more the message that is a problem. It talks about JSON stuff. If I switch to the binary format it will talk about binary problems I guess. The message should be that the serialized data is not matching the expected type.
User avatar
Joel
Moodkie Staff
Posts: 4826
Joined: Wed Nov 07, 2012 10:32 pm

Re: How to handle type changes?

Post by Joel »

Hi there,

In this case I would change the ES3Type so that when writing the property, it uses a different property name. This will allow you to differentiate between the save data stored, and allow you to put in two different save cases depending on the property name. I.e.

Code: Select all

protected override void WriteObject(object obj, ES3Writer writer)
{
	var instance = (MyClass)obj;
	writer.Write("myPropertyNew", instance.myProperty);
}

protected override void ReadObject<T>(ES3Reader reader, object obj)
{
	var instance = (System.Random)obj;
	foreach(string propertyName in reader.Properties)
	{
		switch(propertyName)
		{
					
			case "myProperty":
                var loadedProperty = reader.Read<OldPropertyType>();
                // Do whatever you need to do with it here.
                break;
			case "MyPropertyNew":
                instance.myProperty = reader.Read<NewPropertyType>();
                break;
			default:
				reader.Skip();
				break;
		}
	}
}
All the best,
Joel
Joel @ Moodkie Interactive
Purchase Easy Save | Contact | Guides | Docs | Getting started
Manufacture43-Daniel
Posts: 6
Joined: Wed Sep 09, 2020 9:50 am

Re: How to handle type changes?

Post by Manufacture43-Daniel »

That's an acceptable workaround and in the lines of we actually did already ahah.

However, that means that we have to switch to a fully manual serialization of the members which is something we were exactly to try avoiding. It would be nicer if we could have the default behavior for everything but just those "two" variables...
User avatar
Joel
Moodkie Staff
Posts: 4826
Joined: Wed Nov 07, 2012 10:32 pm

Re: How to handle type changes?

Post by Joel »

As this is the first time this has been requested, I've added a feature request for this here to see if there's demand for this. Ultimately this would be at the expense of performance and file size.

All the best,
Joel
Joel @ Moodkie Interactive
Purchase Easy Save | Contact | Guides | Docs | Getting started
Manufacture43-Daniel
Posts: 6
Joined: Wed Sep 09, 2020 9:50 am

Re: How to handle type changes?

Post by Manufacture43-Daniel »

I fail to see how that would impact file size. The way I see it: the parser will read the data, try to apply it to the values, notice it doesn't fit, buffer the read data and continue. At the end of the parsing, a callback would be called with the buffered data so that the client can choose to make something out of it or not. Performance and memory impact would be minimal.
User avatar
Joel
Moodkie Staff
Posts: 4826
Joined: Wed Nov 07, 2012 10:32 pm

Re: How to handle type changes?

Post by Joel »

Hi there,

The issue is that parsers can't read the data or understand whether there is a mismatch in types without knowing what type the data is (which is why this would need to be stored with the data), and cannot load the data without knowing what type it is. It would also need to load that data into an intermediary variable instead of loading it directly into the field, which essentially doubles memory usage in that process.

If you don't save the type with the data, it's not possible to tell whether the issue due to a mismatch of data types, or whether the data is in the incorrect format or corrupt. The only thing it's possible to know is that the character we were expecting is not the character we encountered (which is the error message we currently throw).

All the best,
Joel
Joel @ Moodkie Interactive
Purchase Easy Save | Contact | Guides | Docs | Getting started
Manufacture43-Daniel
Posts: 6
Joined: Wed Sep 09, 2020 9:50 am

Re: How to handle type changes?

Post by Manufacture43-Daniel »

I didn't check how the binary parser works, but in JSON, there's more or less 3 big types of data: values (string, int, float...), arrays and dictionaries/objects. The differentiation is easy to make since they just start with different characters. You obviously know all that already since this is what started this thread: expecting to read some value and ending up receiving the beginning of an object.

As it is right now, if anybody changes a struct to a dictionary<string, object>, it would probably load just fine right? That's exactly my point here. The strings are already being loaded into memory, it's just a matter of putting them unparsed (unparsed as in not converted to int or float) in dictionaries and lists that would be sent to a callback with the name of their variable to see if the client want to make anything of them. If you don't want to accumulate and wait for the end of the full loading, as this would have a bigger impact on memory, I understand, just call the callbacks as data is gathered.

If I get the time, I'll implement the change myself and put the diffs here.
Post Reply