Page 1 of 2

How to handle type changes?

Posted: Wed Sep 09, 2020 9:59 am
by Manufacture43-Daniel
Hello,

Recently we made a change to one of our saved data structures: a List<Loot> was changed to Dictionary<string, int>. Loot was a structure with several values half of them were not actually useful in the save... Anyway, this change yielded two issues:

- It raised an exception while loading that was very confusing:

Code: Select all

System.FormatException: Expected '[' or "null", found '{'.
As I could gather from stepping through ES3 code, the loader tries to read the data the way the code is formatted instead of reading data first and then applying it to the loaded fields. If it was done the second way, it would be possible to yield a more sensible error message like "Invalid Data Format" or something.

- I didn't find any way of handling data type changes in ES3. What we did was rename the value to another name so that the loading doesn't fail but that also means that all the data ends up empty in that value... It would be nice to have something equivalent to ISerialization's special constructor that takes in a SerializationInfo and a StreamingContext so that we can manually read data and support old data formats. Is there any way of doing so right now that I may have missed?

Re: How to handle type changes?

Posted: Wed Sep 09, 2020 10:49 am
by Joel
Hi there, and thanks for the feedback.
As I could gather from stepping through ES3 code, the loader tries to read the data the way the code is formatted instead of reading data first and then applying it to the loaded fields. If it was done the second way, it would be possible to yield a more sensible error message like "Invalid Data Format" or something.
We do it this way as it significantly improves performance, otherwise you're essentially doubling your memory overhead. Just to clarify, we follow .NET's conventions when it comes to handling incorrectly formatted data (i.e. by throwing a FormatException).
- I didn't find any way of handling data type changes in ES3. What we did was rename the value to another name so that the loading doesn't fail but that also means that all the data ends up empty in that value... It would be nice to have something equivalent to ISerialization's special constructor that takes in a SerializationInfo and a StreamingContext so that we can manually read data and support old data formats. Is there any way of doing so right now that I may have missed?
We allow you to create ES3Types to do this, which are documented here:
https://docs.moodkie.com/easy-save-3/es ... g-es3types

All the best,
Joel

Re: How to handle type changes?

Posted: Wed Sep 09, 2020 10:54 am
by Manufacture43-Daniel
I fail to see how implementing an ES3Type here would save us. The code could be something in the lines of reading the variable in its current data format in a try block and then trying to read the old data format in the catch block. But since the stream would have already been consumed by the first try (and actually left in a pretty weird state) I don't think that would work.

Re: How to handle type changes?

Posted: Wed Sep 09, 2020 12:17 pm
by Manufacture43-Daniel
Also, I'm not exactly against the type of exception being thrown, it's more the message that is a problem. It talks about JSON stuff. If I switch to the binary format it will talk about binary problems I guess. The message should be that the serialized data is not matching the expected type.

Re: How to handle type changes?

Posted: Wed Sep 09, 2020 5:28 pm
by Joel
Hi there,

In this case I would change the ES3Type so that when writing the property, it uses a different property name. This will allow you to differentiate between the save data stored, and allow you to put in two different save cases depending on the property name. I.e.

Code: Select all

protected override void WriteObject(object obj, ES3Writer writer)
{
	var instance = (MyClass)obj;
	writer.Write("myPropertyNew", instance.myProperty);
}

protected override void ReadObject<T>(ES3Reader reader, object obj)
{
	var instance = (System.Random)obj;
	foreach(string propertyName in reader.Properties)
	{
		switch(propertyName)
		{
					
			case "myProperty":
                var loadedProperty = reader.Read<OldPropertyType>();
                // Do whatever you need to do with it here.
                break;
			case "MyPropertyNew":
                instance.myProperty = reader.Read<NewPropertyType>();
                break;
			default:
				reader.Skip();
				break;
		}
	}
}
All the best,
Joel

Re: How to handle type changes?

Posted: Wed Sep 09, 2020 5:34 pm
by Manufacture43-Daniel
That's an acceptable workaround and in the lines of we actually did already ahah.

However, that means that we have to switch to a fully manual serialization of the members which is something we were exactly to try avoiding. It would be nicer if we could have the default behavior for everything but just those "two" variables...

Re: How to handle type changes?

Posted: Wed Sep 09, 2020 6:13 pm
by Joel
As this is the first time this has been requested, I've added a feature request for this here to see if there's demand for this. Ultimately this would be at the expense of performance and file size.

All the best,
Joel

Re: How to handle type changes?

Posted: Thu Sep 10, 2020 9:42 am
by Manufacture43-Daniel
I fail to see how that would impact file size. The way I see it: the parser will read the data, try to apply it to the values, notice it doesn't fit, buffer the read data and continue. At the end of the parsing, a callback would be called with the buffered data so that the client can choose to make something out of it or not. Performance and memory impact would be minimal.

Re: How to handle type changes?

Posted: Thu Sep 10, 2020 10:23 am
by Joel
Hi there,

The issue is that parsers can't read the data or understand whether there is a mismatch in types without knowing what type the data is (which is why this would need to be stored with the data), and cannot load the data without knowing what type it is. It would also need to load that data into an intermediary variable instead of loading it directly into the field, which essentially doubles memory usage in that process.

If you don't save the type with the data, it's not possible to tell whether the issue due to a mismatch of data types, or whether the data is in the incorrect format or corrupt. The only thing it's possible to know is that the character we were expecting is not the character we encountered (which is the error message we currently throw).

All the best,
Joel

Re: How to handle type changes?

Posted: Fri Sep 11, 2020 8:13 am
by Manufacture43-Daniel
I didn't check how the binary parser works, but in JSON, there's more or less 3 big types of data: values (string, int, float...), arrays and dictionaries/objects. The differentiation is easy to make since they just start with different characters. You obviously know all that already since this is what started this thread: expecting to read some value and ending up receiving the beginning of an object.

As it is right now, if anybody changes a struct to a dictionary<string, object>, it would probably load just fine right? That's exactly my point here. The strings are already being loaded into memory, it's just a matter of putting them unparsed (unparsed as in not converted to int or float) in dictionaries and lists that would be sent to a callback with the name of their variable to see if the client want to make anything of them. If you don't want to accumulate and wait for the end of the full loading, as this would have a bigger impact on memory, I understand, just call the callbacks as data is gathered.

If I get the time, I'll implement the change myself and put the diffs here.