Binary Formatting

Vote for new features, or make your own requests here.
Post Reply

Binary formatting

Yes, I would like this feature
13
100%
No, I would not like this feature
0
No votes
 
Total votes: 13

User avatar
Joel
Moodkie Staff
Posts: 4826
Joined: Wed Nov 07, 2012 10:32 pm

Binary Formatting

Post by Joel »

Status

Requested

Complexity

8/10

Description

Support binary formatting.

Note that it's a misconception that binary serialization is inherently more compact or faster than other methods; it's actually the unstructured nature of BinaryWriters/Readers which makes them more compact (i.e storing data sequentially in an inflexible way rather than using formatting).

Using binary with formatted data is unlikely to provide any significant gains in performance or reduction in file size over JSON, and in almost all cases we recommend using Compression to achieve the same thing instead as binary formatting is difficult to debug, not editable in a text editor and inflexible:

https://docs.moodkie.com/easy-save-3/es ... ompression
Joel @ Moodkie Interactive
Purchase Easy Save | Contact | Guides | Docs | Getting started
Michal.Stangel
Posts: 10
Joined: Mon Mar 18, 2019 3:12 pm

Re: Binary Formatting

Post by Michal.Stangel »

Hello,

Are there any plans to implement this feature? I understand it's quite complex and time consuming, but what about some soft version, like replacing most common types with some short codes? Some two digit code instead of "System.Int32", translating it in input and output from dictionary and put this option into ES3Settings. It would reduce size roughly by 20% in most of the cases, which is nice.
User avatar
Joel
Moodkie Staff
Posts: 4826
Joined: Wed Nov 07, 2012 10:32 pm

Re: Binary Formatting

Post by Joel »

Hi there,

We've currently been experimenting with binary formatting but have decided to put it on hold. The reason for this is because we performed some user research to understand why people were asking for binary formatting.

Generally people have been requesting it for three reasons:
  1. Smaller file sizes
  2. To obfuscate save data
  3. To work with APIs which require a byte array
Regarding the first reason: In the majority of cases, the reduction afforded by binary formatting isn't enough to make a difference (a 20% reduction is a superficial amount for most, and in our tests binary formatting doesn't even reduce it by this much). Instead we've implemented compression, which reduces file size by 85% on average.

Regarding the second reason: this is the purpose of encryption. Binary formatting only gives the false sense of obfuscation. We might consider adding XOR obfuscation if people find that AES isn't performant enough.

Regarding the third reason: JSON save files can be converted to byte arrays quite easily using ES3.LoadRawBytes.

We then looked at the cons of binary formatting, which we're very aware of as Easy Save 2 revolved around binary formatting. The main cons being:
  1. It's extremely difficult to debug compared to text-based formats
  2. It's very easy to corrupt. For example, anything which tries to apply an encoding to it will corrupt it, including text editors or incorrectly configured cloud services
  3. Binary formatting is largely proprietary, so it would be impossible to parse the data using other services, making it much less useful
  4. It increases the amount of testing required exponentially
With these in mind, we realised that adding binary formatting wasn't actually adding value to the customers and that these needs could be better addressed in other ways. Hope this helps to describe our justification.

All the best,
Joel
Joel @ Moodkie Interactive
Purchase Easy Save | Contact | Guides | Docs | Getting started
Michal.Stangel
Posts: 10
Joined: Mon Mar 18, 2019 3:12 pm

Re: Binary Formatting

Post by Michal.Stangel »

Hello Joel,

Thanks for detailed explanation, I understand that.
But maybe you could still consider my suggestion in post above. I'm asking, because save files in my RPG game go easily above 20MB and I'm quite woried that average player can produce 100+ saves (there are also auto saves during game), which is 2GB+ and that's huge.

I don't save object or anything more complex, mostly only integers, strings, floats, bools, lists and dictionaries. Basically I have to save a lot of variables for 1000+ NPCs. So shortening some strings in saved JSON would make significant difference in size.

PS: I know there is a compression available, but don't want to impact performance. Will test it though.
User avatar
Joel
Moodkie Staff
Posts: 4826
Joined: Wed Nov 07, 2012 10:32 pm

Re: Binary Formatting

Post by Joel »

Hi there,

Give compression a go as Gzip doesn't impact performance considerably, especially when you use caching. And with larger files you often get some performance back from not needing to write as much data to disk.

With regards to creating shortcuts for type strings, this isn't too difficult to implement at our end. If you private message me your invoice number I can send over a package which does this.

All the best,
Joel
Joel @ Moodkie Interactive
Purchase Easy Save | Contact | Guides | Docs | Getting started
Michal.Stangel
Posts: 10
Joined: Mon Mar 18, 2019 3:12 pm

Re: Binary Formatting

Post by Michal.Stangel »

Joel wrote: Wed Oct 07, 2020 8:29 am With regards to creating shortcuts for type strings, this isn't too difficult to implement at our end. If you private message me your invoice number I can send over a package which does this.

All the best,
Joel
Thank you Joel, but no need to sending me an update as I am not in hurry (long development ahead). Can wait for some future update.
Post Reply