Saving thousands of objects

Discussion and help for Easy Save 3
Post Reply
Radu392
Posts: 2
Joined: Fri May 25, 2018 11:46 pm

Saving thousands of objects

Post by Radu392 »

Hello, I'm wondering what would be the best approach to save (tens) of thousands of objects. Here is some context:

I'm creating a procedurally generated map at runtime, using Gaia and Gena to do the heavy lifting. That's a 2 by 2 km terrain, with thousands of trees on it (as prefabs, not part of the terrain trees), hundreds of rocks, animals and other AI and plenty of other gameobjects. So basically, 99.9% of things are instantiated at runtime. Oh and the navmesh as well, which took quite a few days of headaches to create it in a smart and efficient way. Surprisingly, the whole loading and instantiating part doesn't take longer than a minute, which is an acceptable time frame (except the navmesh, which is extended over a longer period of time in a background thread). So thanks to culling and other little tricks, everything is a smooth sail...

However, save loading is a much more complicated affair. What I tried so far:

I used an asset called AllSavePro, and although that thing worked, it took a whole minute to do a simple autosave and I'm not even talking about loading. Another huge disadvantage it had was that it modified every single script in the project who needed saving and I didn't like that.

So now I tried Easy Save 3 because it looks like it's simpler to select what you need to save exactly, it doesn't modify your own scripts and is just more versatile in general. However I spent 2 days now trying to optimize it as much as I could but to no avail. I tried using the autosave feature but that thang just saved wayyy too much stuff, despite me selecting as few types as possible for save. Then I tried to do it manually to a cached ES3File. But the problem is still there, there is just too much stuff to save. I tried using the ES3 prefab component, but that multiplies the instantiation of my objects by 10000x because Unity always seems to freeze, so I can't use that either.

So I'm wondering, what would be the best way to save say 25'000 gameobjects in less than 3 seconds? I'm thinking one possible solution is to obviously not save EVERYTHING, but only the things that have changed state since the last save/start of game. Sadly, I have no idea how to do that without adding a whole bunch of code to my already numerous scripts. But if there is really no other way around it, I'll have to go this long, tedious way.

Another option I thought of is to look at how the unity terrain behaves. When you create a unity terrain, everything from its heightmap to its details is saved in a terrain object in your assets folder. So although I spend a lot of time instantiating the details at runtime, loading the terrain in future saves is a breeze. Is it possible to do the same thing with normal gameobjects? I saw ES3 can save to resources, but sadly only in the editor whereas I need it to work in built applications.

Thanks for any suggestions!

Radu
User avatar
Joel
Moodkie Staff
Posts: 4826
Joined: Wed Nov 07, 2012 10:32 pm

Re: Saving thousands of objects

Post by Joel »

Hi there,

You will definitely need to be very selecting about what you save.

I believe the only way you could get close to the performance you're looking for would be to use an ES2Writer to save sequentially using Easy Save 2 (which is still supported). This writes data in binary format without formatting, and is essentially the fastest you can write data to a file.

The code for this would look something like this, though this is only an example and you will need to modify it for your own purposes:
using(var writer = ES2Writer.Create("myFile.es2"))
{
    // Write the number of Transforms we're saving.
    writer.Write(instances.Length);

    // Now write the position of each instance.
    for(int i=0; i<instances.Length; i++)
    {
        writer.Write(instance.position);
        writer.Write(instance.rotation);
       // You can add more Write calls here to write other aspects of your object.
    }

    // Save with the 'false' parameter when we're done.
    writer.Save(false);
}
And then to load, you would do something along the lines of:
// Assume that this array contains the Transforms of our prefab instances, all of the same type.
Transform[] instances;

using(var reader = ES2Reader.Create("myFile.es2"))
{
    // Read the number of Transforms we're loading.
    int count = reader.Read<int>();

    // Now instantiate and load the position for each object from the reader.
    for(int i=0; i<count; i++)
    {
        Transform t = ((GameObject)Instantiate(myPrefab)).transform;
        t.position = reader.Read<Vector3>();
        t.rotation = reader.Read<Quaternion>();
    }
}
Note that with ES2Readers/Writers, you have to write in precisely the same order as you loaded. I recommend reading the guide for more information.

All the best,
Joel
Joel @ Moodkie Interactive
Purchase Easy Save | Contact | Guides | Docs | Getting started
Radu392
Posts: 2
Joined: Fri May 25, 2018 11:46 pm

Re: Saving thousands of objects

Post by Radu392 »

Thank you so much! I read about binaryFormat and the related stuff and how fast it is, but I didn't expect an INSTANT save time. Now saving doesn't even skip a frame. Loading however, takes much longer than I'd like, but that's because of Unity's Instantiate method which is very slow and unrelated to file I/O. I'm sure I'll find a way around that too though.

Thanks again for your help!
User avatar
Joel
Moodkie Staff
Posts: 4826
Joined: Wed Nov 07, 2012 10:32 pm

Re: Saving thousands of objects

Post by Joel »

No problem, glad to be of assistance :)

All the best,
Joel
Joel @ Moodkie Interactive
Purchase Easy Save | Contact | Guides | Docs | Getting started
Post Reply