Sequentially loading and writing with es3 ?

[davidlim]

Hello I saw the feature in es2 but I cant seem to be able to do this in es3 and there is not write method that accepts only a value without a key.

I tried using the ES3File but it seems a bit to slow (clocks at about 1 second to load my data) I'm sure it's fast after that (with the caching) but I need to be able to load these files on demand and fast.
I can split it to a lot of smaller files and load them one by one, but that would be a hassle as I would just rather load sequentially.

I tought about reading raw bytes, if I do this I could use threads, but I'm not sure how to deserialize ES3 data from raw bytes and I saw no mention of it in the docs.

In short:

I have a large world, when the character gets to a certain point I need to start loading some data in order to generate the world around him, that data can be saved (an is) in multiple files, currently it would be around 1000 files ( a file per area) so splitting it up even more seems like hell.
I would appreciate any ideas or thoughts for stuff I might have missed.

[Joel]

Hi there,

When are you creating your ES3File? The idea with caching is that you load your data into an ES3File when the scene loads, and then load from this ES3File from there on. It sounds like you might be creating a new ES3File every time you load data, so then you're encountering the overhead of loading the data from file each time. Is this the case?

Note that if you have a lot of data to load, then the only way to improve performance will be to pre-load the data, or break the loading over several frames (this won't make it any faster per se, but it will stop the main thread from becoming blocked).

With regards to threading, you can use Easy Save in a separate thread, but because much of Unity's API is not thread-safe, you will need to ensure that the data you're saving/loading is in a suitable state.

Would you be able to show me the code you're using to save and load? There may be some way of saving less data.

All the best,
Joel

[davidlim]

Hello Joel, thanks for the answer

I am creating the ES3File instance when I load the data as the data needs to be loaded on scene load, my game has a bunch of cells each cell is a scene, if the player gets near the edge of the scene I load the next scene using loadSceneAsync, then on the awake method I load the data(by creating a ES3File) and then then load parts of the data every frame.

As the game Is an open world game I cannot preload all of the data, I would like to load part of the data each frame the issue is that ES3File cannot be loaded in parts, once I create it, it loads all of the data in one chunk.

The save method looks something like this:
it only happens in the editor so I dont really care about how fast it is right now.

Code: Select all

                
                var file = new ES3File($"${gameObject.scene.name}.grid.${tilemap.name}");
                foreach (var position in tilemap.cellBounds.allPositionsWithin)
                {
                    if (tilemap.HasTile(position))
                    {
                        var tile = (Tile) tilemap.GetTile(position);
                        file.Save<WorldTile>(((Vector2Int)position).ToString(), WorldTile.Create(tile, position));
                        spriteSet.Add(tile.sprite);
                    }
                }                
                
                file.Sync();

The load method is just creating a new ES3File instance of the same name as above on the scene load.

Code: Select all

                for (var index = 0; index < bounds.Count; index++)
                {
                    var position = bounds[index];
                    var key = position.ToString();

                    var tile = file.Load<WorldTile>(key, null);
                    if (tile != null)
                    {
                        tilemap.SetTile(tile.Position, tile.ToTile());
                    }

                    if (index % 100 == 0)
                    {
                        yield return new WaitForEndOfFrame();                        
                    }

As far as I tested the actual time consuming line is the creation of the File for loading.

[Joel]

Hi there,

In this case the only alternative would be to split it into smaller files, as the overhead will be the loading of the data.

You could also try going to Window > Easy Save 3 > Settings, open Advanced Runtime Settings and increase the size of the buffer, just incase more data can be buffered. And again, you could try running it in a separate thread, assuming that the data you're saving/loading is thread safe.

If it was possible to load sequentially in Easy Save 3 it would not help here because to access data at the end of the file you would still need to buffer all of the data.

All the best,
Joel