QField collects large amounts of data which affects it performance

[ckuipersatkb]

What is the bug or the crash? What were your expectations and what actually happened?

I am not sure if this is the best place for the feedback / question, but I you can still give me some advice.

We have around 50 fieldworkers who are using QField on a daily basis. We have four template projects which are used for almost all our projects. Project preparation consists of copying a template project, drawing the research area and publishing the project to QFieldCloud.

While working in the field, QField is collecting a large amount of data. The data is stored as cache or as user data. The amount can be seen under Android settings > Apps > QField > Storage.

The speed at which the storage grows different for the templates. A template with virtual fields and overlay_intersect() expressions generates more data than a template with only a relation reference which can be filtered through another field. Of course, rendering the base maps also stores data in the cache.

I have not tested this, but my impression is that one or two weeks the accumulated cache starts to affect QField's performance. Within this period, the cache can be built up to 600Mb-1,5 Gb. When working on multiple projects in a week, the user data can also grow to 500-800 Mb (or more).
The storage of QField with cache, user data and app size can easily reach to 3,5Gb.

When the amount of collected data becomes large,we experience some issues, for example:

• Searching or loading the dropdown list of a relation reference becomes slower or the list does not work at all,
• Base maps are not visible any more
• Layers are not visible while there is no error message about broken link or something
• In one case, a 500mb project could not be downloaded. After emptying cache and deleting the user data, the project downloaded succesfully.

My questions are:

• Is it already known that clearing the cache and deleting the app data resolves a number of issues? Is the accumulated cache/user data likely to be the cause?
• Is there a recommended way to limit the build-up of cache and user data?

Steps to reproduce the issue

Version

All of them. Currently, 4.1.12.

Operating system name

Android

Operating system version

16

Reinstall QField

• I have a fresh install of the latest QField version, but the problem persists.
• Problem can be reliably reproduced, doesn't happen randomly.
• Problem happens with all files and projects, not only some files or projects.

Additional context

Since this happens with all projects I am not sure about sending an example project.

[nirvn]

@ckuipersatkb , good day.

Your issue intrigues me :) with regards to growing cache (i.e. data that would not be stored in the project's datasets themselves), this would be a growth tied to accumulated download of XYZ raster and/or vector tiles. Is that what you are referring to? This part I'm not sure I understand:

A template with virtual fields and overlay_intersect() expressions generates more data than a template with only a relation reference which can be filtered through another field.

What do you mean by that?

[ckuipersatkb]

Hey @nirvn,

Yes, the cache is tied to the XYZ rasters and/or vector tiles. But it feels like the cache contains more data than just the base maps.

For the virtual fields and overlay_intersect() expression, you can use ATKB_TE/TE_20260018_SNL_FLORA as a example project. A fieldworker tracking their position. A grid-layer is turning blue when the track is intersecting a grid cell. This is doen by a virual field with an overlay_intersect() expression.

When the map is refreshing, the virtual field is recalculated. Is this data stored in the cache?

In the exmaple project, we have an observation layer (Waarnemingen). The observed species is filled in in the field SOORT. This is a relation reference-widget to the layer "soortenlijst" which is filtered based on other fields (TYPE_WAARNEMING and SOORTGROEP).
To make it more complex, there are two tables related to the species list (soortenlijst). One contains the favorite species of a fieldworker (based on @user_account_name) and one contains the target species of a project. The favorite and target lists are actually stored within the complete species list. This setup is used so the observatios-layer (Waarnemingen) contains just one species field, (When analysing the data, our ecologists were confused by multiple species fields. )

Are the filtering and refreshing of the species lists stored in the cache?

Does this make it clearer?

[nirvn]

When the map is refreshing, the virtual field is recalculated. Is this data stored in the cache?

Definitively not. The accumulation of on-disk cache would only come from your online raster and vector tiles layers.

[ckuipersatkb]

@nirvn I am a little bit stubborn but a created a project without online base maps to test this: ATKB_TE/TE_TEST_CACHE_SNL_INSECTEN/

Before downloading the project I cleared the cache and the user data.
Opening QField afterwards results in:

• Cache=3,69Mb
• User data = 23,76MB.

After 23 minutes, 26 changes and tracking my position:

• Cache=14,0Mb
• User data=39,9MB

Taking pictures, adding features and pushing the changes all affected the cache and User data. After opening the project the cache was relatively stabile. The user data was increasing with almost every action. Mostly with 0,1MB but at the end this will accumulate to quite a lot.

[nirvn]

@ckuipersatkb , nice stats. Let's try to dig deeper then since you seem keen :)

On the user data, can we identify which files are growing and amount to a ~16.2mb increase? Does that include images taken?

[ckuipersatkb]

@nirvn
I have check a few files and I cannot find the 16,2Mb.

Taken a picture did cause an increase in User data, but I am not sure if the increase in storage is the same as the file size. I don't think so.
In total the pictures are 6,5MB

I recreated the project to check the data.gpkg This is a difference of 0,03MB

Synchronizing the project to QGIS caused an increase of 27,3Mb (incl the picture). So, an increase of 20,8 MB without the pictures.

The folders in the .qfiedlsync diroctry don't match the 16,2MB

• backup = 15,7Mb (close, but will be weird since it contains the layers with changes)
• download = 22,2MB
• uplaoad -> empty after syncing to QGIS

[ckuipersatkb]

@nirvn The release of QField 4.3.0 sounds promising. Thanks for the hard work during the summer of stability to all your colleagues.

In the release notes the stability and “speed up data entry” are mentioned. Did you find anything about the accumulation of the cache and user data? And are some changes made concerning this issue?