Hello, I'm wondering what would be the best approach to save (tens) of thousands of objects. Here is some context:
I'm creating a procedurally generated map at runtime, using Gaia and Gena to do the heavy lifting. That's a 2 by 2 km terrain, with thousands of trees on it (as prefabs, not part of the terrain trees), hundreds of rocks, animals and other AI and plenty of other gameobjects. So basically, 99.9% of things are instantiated at runtime. Oh and the navmesh as well, which took quite a few days of headaches to create it in a smart and efficient way. Surprisingly, the whole loading and instantiating part doesn't take longer than a minute, which is an acceptable time frame (except the navmesh, which is extended over a longer period of time in a background thread). So thanks to culling and other little tricks, everything is a smooth sail...
However, save loading is a much more complicated affair. What I tried so far:
I used an asset called AllSavePro, and although that thing worked, it took a whole minute to do a simple autosave and I'm not even talking about loading. Another huge disadvantage it had was that it modified every single script in the project who needed saving and I didn't like that.
So now I tried Easy Save 3 because it looks like it's simpler to select what you need to save exactly, it doesn't modify your own scripts and is just more versatile in general. However I spent 2 days now trying to optimize it as much as I could but to no avail. I tried using the autosave feature but that thang just saved wayyy too much stuff, despite me selecting as few types as possible for save. Then I tried to do it manually to a cached ES3File. But the problem is still there, there is just too much stuff to save. I tried using the ES3 prefab component, but that multiplies the instantiation of my objects by 10000x because Unity always seems to freeze, so I can't use that either.
So I'm wondering, what would be the best way to save say 25'000 gameobjects in less than 3 seconds? I'm thinking one possible solution is to obviously not save EVERYTHING, but only the things that have changed state since the last save/start of game. Sadly, I have no idea how to do that without adding a whole bunch of code to my already numerous scripts. But if there is really no other way around it, I'll have to go this long, tedious way.
Another option I thought of is to look at how the unity terrain behaves. When you create a unity terrain, everything from its heightmap to its details is saved in a terrain object in your assets folder. So although I spend a lot of time instantiating the details at runtime, loading the terrain in future saves is a breeze. Is it possible to do the same thing with normal gameobjects? I saw ES3 can save to resources, but sadly only in the editor whereas I need it to work in built applications.
Thanks for any suggestions!
Radu
Saving thousands of objects
Re: Saving thousands of objects
Hi there,
You will definitely need to be very selecting about what you save.
I believe the only way you could get close to the performance you're looking for would be to use an ES2Writer to save sequentially using Easy Save 2 (which is still supported). This writes data in binary format without formatting, and is essentially the fastest you can write data to a file.
The code for this would look something like this, though this is only an example and you will need to modify it for your own purposes:
All the best,
Joel
You will definitely need to be very selecting about what you save.
I believe the only way you could get close to the performance you're looking for would be to use an ES2Writer to save sequentially using Easy Save 2 (which is still supported). This writes data in binary format without formatting, and is essentially the fastest you can write data to a file.
The code for this would look something like this, though this is only an example and you will need to modify it for your own purposes:
using(var writer = ES2Writer.Create("myFile.es2")) { // Write the number of Transforms we're saving. writer.Write(instances.Length); // Now write the position of each instance. for(int i=0; i<instances.Length; i++) { writer.Write(instance.position); writer.Write(instance.rotation); // You can add more Write calls here to write other aspects of your object. } // Save with the 'false' parameter when we're done. writer.Save(false); }And then to load, you would do something along the lines of:
// Assume that this array contains the Transforms of our prefab instances, all of the same type. Transform[] instances; using(var reader = ES2Reader.Create("myFile.es2")) { // Read the number of Transforms we're loading. int count = reader.Read<int>(); // Now instantiate and load the position for each object from the reader. for(int i=0; i<count; i++) { Transform t = ((GameObject)Instantiate(myPrefab)).transform; t.position = reader.Read<Vector3>(); t.rotation = reader.Read<Quaternion>(); } }Note that with ES2Readers/Writers, you have to write in precisely the same order as you loaded. I recommend reading the guide for more information.
All the best,
Joel
Re: Saving thousands of objects
Thank you so much! I read about binaryFormat and the related stuff and how fast it is, but I didn't expect an INSTANT save time. Now saving doesn't even skip a frame. Loading however, takes much longer than I'd like, but that's because of Unity's Instantiate method which is very slow and unrelated to file I/O. I'm sure I'll find a way around that too though.
Thanks again for your help!
Thanks again for your help!
Re: Saving thousands of objects
No problem, glad to be of assistance
All the best,
Joel
All the best,
Joel