Intermittent crashes when saving data to Firestore

[khambadkone]

[REQUIRED] Please fill in the following fields:

• Unity editor version: 2020.3.32f1
• Firebase Unity SDK version: 8.8.0
• Source you installed the SDK: .unitypackage
• Problematic Firebase Component: Firestore
• Other Firebase Components in use: Firestore, Auth
• Additional SDKs you are using: IronSource, mutiiple ad network SDKs
• Platform you are using the Unity editor on: Mac
• Platform you are targeting: Android
• Scripting Runtime: IL2CPP
• Pre-built SDK from the website or open-source from this repo: Pre-built

[REQUIRED] Please describe the question here:

Using a combination of Firebase Authenticate and Firestore, we save player information as Documents to a collection on Firestore. This data is a set of key-value data pairs, indicating player progress and the resulting document hierarchy is flat.

The game uses Firebase 8.8.0 with Unity 2020.3.32f1. The Unity code that saves this data is :-

public override void Save<T>(T saveData, OnSaveProgress onSaveComplete)
        {
            Logger.Log($"FireStoreCloudSave saving... {id}");
            GetDocumentReference().SetAsync(saveData).ContinueWith(t =>
            {
                if (t.Exception != null)
                {
                    Logger.Log(t.Exception.ToString());
                }
                else
                {
                    Logger.Log($"save complete {id}");
                }
            });
            onSaveComplete?.Invoke(Type.FIRESTORE);
        }

saveData is a class

[System.Serializable]
public class GameSaveData : SerializableDictionary { }

The code works for 90% of users, but we see the following two crashes in Crashyltics to the remaining 8-10% users :-

Crashlog 1:

(multiple libbase.so Missing UUID xxxxxxxx)
8 libil2cpp.so FirestoreCpp.cs ConvertMapToFieldValue
9 libil2cpp.so ConverterBase.cs ConvertToProxyMap
10 libil2cpp.so DocumentReference.cs SetAsync
11 libil2cpp.so GenericMethods5.cpp - Line 40493 FireStoreCloudSave_Save_TisRuntimeObject_m3EFAD2C4096136226A4FF17AE9A907926CB9744E_gshared + 40493
12 libil2cpp.so GenericMethods6.cpp - Line 89 SaveGameManager_Save_TisRuntimeObject_mD14C102D68CE7A760FC8F9AAE2A83E1C545137A8_gshared + 89

Crashlog 2 :

(multiple libbase.so Missing UUID xxxxxxxx)
8 libil2cpp.so FirestoreCpp.cs DocumentReferenceSetAsync
9 libil2cpp.so GenericMethods5.cpp - Line 40493 FireStoreCloudSave_Save_TisRuntimeObject_m3EFAD2C4096136226A4FF17AE9A907926CB9744E_gshared + 40493
10 libil2cpp.so GenericMethods6.cpp - Line 89 SaveGameManager_Save_TisRuntimeObject_mD14C102D68CE7A760FC8F9AAE2A83E1C545137A8_gshared + 89

On checking our logs and Firestore, we see data available for these users.

Am looking to understand what causes these errors in Crashlytics and the behaviour of Firestore when they occur ?

Crashlog 1 : Is this due to an issue converting a non-compliant data type in Firestore ? FieldValue documentation mentions unsupported types can cause a crash. If so, how can we identify the problem elements in Unity ? Why do we see data in FireStore despite these crashes ?

Crashlog 2 : What can cause a crash on SetAsync ? Is this due to the 1 second limit on saving documents ?

[google-oss-bot]

This issue does not seem to follow the issue template. Make sure you provide all the required information.

[khambadkone]

This issue does not seem to follow the issue template. Make sure you provide all the required information.

Updated.

[paulinon]

Hi @khambadkone,

Thanks for reporting this issue. I can bring this up to the team so that we can identify the causes of these crashes. For now, could you share the details (such as model and OS version) of the devices encountering the crashes?

Additionally, could you confirm if version 9.6.0 reduces that crashes you're facing?

[dconeybe]

This looks like a known issue that just came to our attention last week. If it's the issue I think it is, then it is specific to Android. Googlers can see b/251869890 for more details.

Basically, Firestore's C++ JNI layer creates too many global references and eventually exhausts the 51200 global ref limit, which causes the app to crash. It is a long-standing bug but only now is starting to get noticed. I'm working on a fix, but the fix is non-trivial and ETA is probably mid-to-late November.

Could you try running this code and see if it looks like the same crash you're getting in Crashlytics:

var numbers = new List<object>();
for (int i=0; i<60000; i++) {
  numbers.Add(i);
}
doc.SetAsync(new Dictionary<string, object>{{"numbers", numbers}});

You should see an error like this in the logcat:

JNI ERROR (app bug): global reference table overflow (max=51200)

which is the telltale sign that you've experienced this issue.

This crash is triggered by having documents with a large number of values (e.g. arrays of thousands of elements) or a large number of documents each with a large number of values (e.g. 60 documents each with an array of 100 elements). Given this information, does it sound like your use case could trigger this crash?

[khambadkone]

Thank you for the reply. Appreciate it.

Each Document is flat, but can have a large number of values. They are all key-value pairs only, however, which is a dump of some legacy JSON.
e.g. lots of :

isXXX54Part1Verb:1
rvABCCount:0

Will try mimicking a large dummy document to confirm it breaks at 60000 dummy values.

[dconeybe]

Will try mimicking a large dummy document to confirm it breaks at 60000 dummy values.

Yes. Each key/value pair in a document uses 1 global ref. For map and array values, there is 1 global ref per entry. So calling DocumentReference.Set() with 51200 key/value pairs or with a single key/value pair whose value is a List of 51200 elements would trigger this crash.

To be clear, this is NOT a good thing; the number of global refs should not be growing linearly with the size of the documents. That is the bug I am going to fix.

[khambadkone]

Hi @dconeybe We did some tests ourselves, and find that the crash does not seem to depend only on the number of elements in the Document, but also on the periodicity of saves.

We have the following snippet we use to keep adding more keys to our Document, and find that it triggers a crash even at 12000 unique keys, provided we add 1000 keys at a time. AddKeys is a method that accepts a number of keys to add.

This differs from the behaviour you mentioned, so curious if this was a known issue fixed in a more recent version of Firestore, or if this is a new bug ?

public void AddKeys(InputField keysText)
    {
        int keys = Int32.Parse(keysText.text);
        const string s = "abcdhf_a";
        Debug.Log($"Adding {keys} keys");
        for (var i = 0; i < keys; i++)
        {
            int r = UnityEngine.Random.Range(0, 3);
            switch (r)
            {
                case 0:
                    GameManager.progressionData.SetInt(System.Guid.NewGuid().ToString(), 1);
                    break;
                case 1:
                    GameManager.progressionData.SetString(System.Guid.NewGuid().ToString(), s);
                    break;
                case 2:
                    GameManager.progressionData.SetFloat(System.Guid.NewGuid().ToString(), 5.5f);
                    break;
            }
        }
        keyCount.text = GameManager.progressionData.gameSaveData.Count.ToString();
        string data = JsonUtility.ToJson(GameManager.progressionData.gameSaveData);
        float size = System.Text.ASCIIEncoding.ASCII.GetByteCount(data)/ 1024f / 1024f;
        stringSize.text = size.ToString("n2") + "MB";
        SaveGameManager.Instance.Save(GameManager.progressionData.gameSaveData, true);
    }