[Meta] external apps improvement roadmap

[RedGl0w]

List of things necessary to make external really usefull :

• be able to use ITCM Ram (usefull for noPIE apps and data section) : [External apps] Allow access to ITCM RAM and reduce slot size #1869
• don't duplicate the storage section for both slots : [External apps] Allow access to ITCM RAM and reduce slot size #1869
• avoid suspending when off is pressed, if not stalling (this wouldn't allow the off button to be used for nofrendo NES emulator for instance) : [Ion] Preemtion and circuit breaker changes to allow access to Home and OnOff keys #1874
• safer heapRange (currently, the heapRange function juste take the whole heap section, without asserting that it isn't yet used by the userland)
• in userland ISR, introduce along heapRange and drawString functions to access to userland storage
• introduce a real file system in flash (for apps and other files) : Use external Flash chip for storage #1547
• either use the inegrated arm-none-eabi-gcc relocation flags (-msingle-pic-base -mpic-register=r9 -mno-pic-data-is-text-relative -fPIC), or, which would be nicer, introduce in the kernel a dynamic relocator and linker (in order to avoid these ugly ISR functions)
• move the appsContainer storage next to the heap, so external apps can use all this memory at the same time

[RedGl0w]

• in userland ISR, introduce along heapRange and drawString functions to access to userland storage

Maybe we could take the same functions as zardam did in external, without the tarball fs.

• either use the inegrated arm-none-eabi-gcc relocation flags (-msingle-pic-base -mpic-register=r9 -mno-pic-data-is-text-relative -fPIC), or, which would be nicer, introduce in the kernel a dynamic relocator and linker (in order to avoid these ugly ISR functions)

There are I think 4 ways to have bss / data sections in external apps :

• have static linked apps with a static buffer
• having a PIE app, and using the toolchain flags for self relocating
• having a PIE app, and do a self relocating script in the app
• having a PIE app, and do relocation by the userland, with the possibility for the future to have a dynamic linker, which would be nice to reduce binary sizes, and avoid using magic values for the apps (svc and userland ISR)

[EmilieNumworks]

Regarding the item

in userland ISR, introduce along heapRange and drawString functions to access to userland storage

I would suggest the following API:

int createRecord(const char * baseName, const char * extension, const void * data, size_t size);
int eraseRecord(const char * baseName, const char * extension);
int readRecord(const char * baseName, const char * extension, const void ** dataBuffer, size_t * sizeBuffer);
int numberOfRecordsWithExtension(const char * extension);
int readRecordWithExtensionAtIndex(const char * extension, int index, const void ** dataBuffer, size_t * sizeBuffer);
 /* The returned int could carry the potential error:
  *   None = 0,
  *   NameTaken = 1,
  *   NonCompliantName = 2,
  *   NotEnoughSpaceAvailable = 3,
  *   RecordDoesNotExist = 4,
  *   RecordIsProtected = 5
  */

Would it be convenient/sufficient?

An important question is (@Ecco, @artaxxx): should the third party app access the records used by the userland - Python scripts, functions, sequences etc - or should we reserve some extensions not to be used by the external apps (and always return the error 'RecordIsProtected' if an external app tries to read/write these records?)

[RedGl0w]

Would it be convenient/sufficient?

I don't see yet any functions missing, but I think time will tell us if there are.

An important question is (@Ecco, @artaxxx): should the third party app access the records used by the userland - Python scripts, functions, sequences etc - or should we reserve some extensions not to be used by the external apps (and always return the error 'RecordIsProtected' if an external app tries to read/write these records?)

It seems a bit overkilled (when third party apps are enabled, we won't have access to exam mode until next reset), and moreover, we already have access to userland RAM, and so userland storage without limitiations.

[Ecco]

Here's a cleaned-up version of the API:

typedef enum {
  ION_STORAGE_RECORD_ERROR_NONE = 0,
  ION_STORAGE_RECORD_ERROR_NAME_TAKEN = 1,
  ION_STORAGE_RECORD_ERROR_NON_COMPLIANT_NAME = 2,
  ION_STORAGE_RECORD_ERROR_NOT_ENOUGH_SPACE_AVAILABLE = 3,
  ION_STORAGE_RECORD_ERROR_RECORD_DOES_NOT_EXIST = 4
} ion_storage_record_error_t;

ion_storage_record_error_t ion_storage_record_create(const char * base_name, const char * extension, const void * data, size_t size);
const void * ion_storage_record_get_data(const char * base_name, const char * extension);
size_t ion_storage_record_get_size(const char * base_name, const char * extension);
void ion_storage_record_destroy(const char * base_name, const char * extension);

It doesn't have the ability to find/list records just yet, but this way we avoid having a record type entirely and it makes for a much simpler API 😄