The List is growing a bit too much

[Rudxain]

Intro

As of now, uad_lists is ~1.4MB.

#608 (reply in thread)

Some text editors may have performance problems when loading the file (but let's be honest, those editors are poorly optimized).

Potential solutions

Format

On #516, I proposed migrating from JSON to some other standard format. But maybe we could benefit from a specially-designed format. I could write a parser that not only validates syntax, but also the values of the fields (similar to #536)

I managed to reduce the List to ~0.9MB by:

• removing unnecessary quotes (like YAML does)
• using single-letter keys
• removing all indentation

For reference, compressing the "non-minified" file with

xz -9e -T1 uad_lists.json

yields a ~0.19MB file. While the minified one compresses to ~0.18MB

For simplicity, we could also choose CSV

Split

Aside from modifying the file, we could also break it in multiple files, for better organization.

I would propose 1 file for each major OEM, as that would allow UAD-ng to avoid loading all lists, iff 1-or-more devices from the same OEM are connected. This can also serve as a network optimization, as only 1 OEM list needs to be updated at a time.

#608 (reply in thread)

See also

• meta(pkg-lists): Move uad_lists to a separate repository #612 / Move `uad_lists` to a separate repository #646

[Rudxain]

I've noticed we use spaces for indentation 💀. We really should use tabs, while we decide what to do

[AnonymousWP]

Sounds good for a major future release (e.g. 2.0.0).

[Rudxain]

Yeah. ~1MB isn't really "too much". But if it ever gets to ~4MB we may get into problems, because GH refuses to render "large files"

[samolego]

1 file per OEM seems quite reasonable.

Question about carriers: don't they also sometimes put stuff on the phones (that's not the case for my country, but idk how it is in US / elsewhere). In that case, perhaps also 1 file per carrier, then you choose the phone+carrier combo.

[Rudxain]

Nice idea! I forgot about carriers, lol.

In that case, instead of:

• 1 list per OEM
• An extra list for everything else

We would have:

• 1 per OEM
• 1 per carrier
• Only 1 Google + AOSP. Because sometimes it's unclear which is which, and they can switch between versions.
• 1 for everything else: chipset manufacturers, unknown/unlisted stuff, 3rd-party partners (such as Meta and Amazon), etc...

[AnonymousWP]

How do we want to migrate that though? Nobody's going to do this manually, and not all package names lead to an OEM. Based on descriptions may help, but still perfect.

This is why I prefer 1 file per OEM. Downside is that if we add a new OEM to the list, we need to create a new file and also build-in support for that within UAD-ng.

[Rudxain]

not all package names lead to an OEM.

Good point!

build-in support for that within UAD-ng.

I assume the GH API supports listing directories. So we could put all lists in the same folder, then code UADNG to download all files from that dir

[shuvashish76]

https://github.com/MuntashirAkon/android-debloat-list did it with similar approach. Would be nice have separate lists according to their category.
Edit: See #646 discussion

[samolego]

Also, how about making dependencies, labels and neededBy optional? Would save some space by those apps which pnly have empty arrays there.

[AnonymousWP]

Yes, I think we agreed (and 0x192 also wanted to do this) upon removing these fields, as they're not displayed in UAD-ng anyway as far as I know.

[Rudxain]

I was considering that if a package "A" is a dependency of "B", then removing "A" should auto-rm "B". This behavior would turn UAD-NG into a package manager (it already is, in some way)

[lavafroth]

Ok, here's a more exotic idea. What if these files also double as a knowledge-base? We create a separate repo with markdown files for each category. uad pulls these markdown files, parses them for use. That way, even a passer by will be able to see descriptions of the apps.

Also see: dnscrypt-proxy2 which uses markdown files for public resolvers.

[AnonymousWP]

I do prefer Markdown, but maybe it's more prone to error? If you make a small typo (perhaps syntax), then it can't be parsed. Is there a Rust lib that can parse Markdown stuff then?

[Rudxain]

This info might be helpful. It seems feasible to parse MD, especially if we auto-enforce a subset of it.

BTW, I've been thinking that 1 file per package might actually be a good idea, as it significantly reduces Git conflicts, and makes it easier to search stuff. We can still have "author folders" for each OEM, carrier, etc...

[AnonymousWP]

It's probably a lot slower, because it'll have to fetch all the separate files. But yeah, I'll only seriously look into these things after UAD-ng has reached adulthood. 🙂

[Rudxain]

It's probably a lot slower

True, because of multiple HTTP requests. Unless we use this "hack" (it's manual, but there must be a way to automate it)

I'll only seriously look into these things after UAD-ng has reached adulthood. 🙂

I agree, it's a bit too soon for this

[Rudxain]

hold up, if we move the list to a separate repo, then we can download all files in a single request!

[est]

can we split the big file into Package Names directly?

like com/android/xxx.json then apps/services can clone and compile it into internal formats.