Saving thousands of objects
Posted: Sat May 26, 2018 12:27 am
Hello, I'm wondering what would be the best approach to save (tens) of thousands of objects. Here is some context:
I'm creating a procedurally generated map at runtime, using Gaia and Gena to do the heavy lifting. That's a 2 by 2 km terrain, with thousands of trees on it (as prefabs, not part of the terrain trees), hundreds of rocks, animals and other AI and plenty of other gameobjects. So basically, 99.9% of things are instantiated at runtime. Oh and the navmesh as well, which took quite a few days of headaches to create it in a smart and efficient way. Surprisingly, the whole loading and instantiating part doesn't take longer than a minute, which is an acceptable time frame (except the navmesh, which is extended over a longer period of time in a background thread). So thanks to culling and other little tricks, everything is a smooth sail...
However, save loading is a much more complicated affair. What I tried so far:
I used an asset called AllSavePro, and although that thing worked, it took a whole minute to do a simple autosave and I'm not even talking about loading. Another huge disadvantage it had was that it modified every single script in the project who needed saving and I didn't like that.
So now I tried Easy Save 3 because it looks like it's simpler to select what you need to save exactly, it doesn't modify your own scripts and is just more versatile in general. However I spent 2 days now trying to optimize it as much as I could but to no avail. I tried using the autosave feature but that thang just saved wayyy too much stuff, despite me selecting as few types as possible for save. Then I tried to do it manually to a cached ES3File. But the problem is still there, there is just too much stuff to save. I tried using the ES3 prefab component, but that multiplies the instantiation of my objects by 10000x because Unity always seems to freeze, so I can't use that either.
So I'm wondering, what would be the best way to save say 25'000 gameobjects in less than 3 seconds? I'm thinking one possible solution is to obviously not save EVERYTHING, but only the things that have changed state since the last save/start of game. Sadly, I have no idea how to do that without adding a whole bunch of code to my already numerous scripts. But if there is really no other way around it, I'll have to go this long, tedious way.
Another option I thought of is to look at how the unity terrain behaves. When you create a unity terrain, everything from its heightmap to its details is saved in a terrain object in your assets folder. So although I spend a lot of time instantiating the details at runtime, loading the terrain in future saves is a breeze. Is it possible to do the same thing with normal gameobjects? I saw ES3 can save to resources, but sadly only in the editor whereas I need it to work in built applications.
Thanks for any suggestions!
Radu
I'm creating a procedurally generated map at runtime, using Gaia and Gena to do the heavy lifting. That's a 2 by 2 km terrain, with thousands of trees on it (as prefabs, not part of the terrain trees), hundreds of rocks, animals and other AI and plenty of other gameobjects. So basically, 99.9% of things are instantiated at runtime. Oh and the navmesh as well, which took quite a few days of headaches to create it in a smart and efficient way. Surprisingly, the whole loading and instantiating part doesn't take longer than a minute, which is an acceptable time frame (except the navmesh, which is extended over a longer period of time in a background thread). So thanks to culling and other little tricks, everything is a smooth sail...
However, save loading is a much more complicated affair. What I tried so far:
I used an asset called AllSavePro, and although that thing worked, it took a whole minute to do a simple autosave and I'm not even talking about loading. Another huge disadvantage it had was that it modified every single script in the project who needed saving and I didn't like that.
So now I tried Easy Save 3 because it looks like it's simpler to select what you need to save exactly, it doesn't modify your own scripts and is just more versatile in general. However I spent 2 days now trying to optimize it as much as I could but to no avail. I tried using the autosave feature but that thang just saved wayyy too much stuff, despite me selecting as few types as possible for save. Then I tried to do it manually to a cached ES3File. But the problem is still there, there is just too much stuff to save. I tried using the ES3 prefab component, but that multiplies the instantiation of my objects by 10000x because Unity always seems to freeze, so I can't use that either.
So I'm wondering, what would be the best way to save say 25'000 gameobjects in less than 3 seconds? I'm thinking one possible solution is to obviously not save EVERYTHING, but only the things that have changed state since the last save/start of game. Sadly, I have no idea how to do that without adding a whole bunch of code to my already numerous scripts. But if there is really no other way around it, I'll have to go this long, tedious way.
Another option I thought of is to look at how the unity terrain behaves. When you create a unity terrain, everything from its heightmap to its details is saved in a terrain object in your assets folder. So although I spend a lot of time instantiating the details at runtime, loading the terrain in future saves is a breeze. Is it possible to do the same thing with normal gameobjects? I saw ES3 can save to resources, but sadly only in the editor whereas I need it to work in built applications.
Thanks for any suggestions!
Radu