Update STM32F4 HAL to include latest official HAL from ST

[josesimoes]

Although working the existing headers for the STM32F4 are outdated. This is a problem when creating new solutions based in STM32F4 devices other than the '407. "Marrying" with current CMSIS headers and code can be also be difficult.

I have a branch with the latest official HAL from ST.
The HAL code files were updated were required (mostly UART, SPI and PWM because there are diferences on what modules are available on each device).
The HAL projects were cleaned up.
The templates were updated.
The platform_selector.h in the solutions was updated to use the new device identifier.
A new file with the HAL configuration was added to the solutions.

[josesimoes]

@smaillet-ms risking asking the obvious:
Assuming that the approach implemented in Llilum was done learning from what might be wrong/not so good in NETMF, can it be implemented in NETMF too? Or is that too complicated or requires a great deal of changes?

[smaillet-ms]

Yes, Llilum was built with full awareness of things like mBed and CMSIS, which didn't exist when NETMF was created. The goal was to minimize the amount of HAL like code we define or standardize around. One of the challenges with NETMF is the rather blurry lines between HAL, PAL, CPU and CLR code layering. With Llilum things are more clear because of the C# vs C/C++ split. So, what I think we need to do with NETMF is migrate in stages:

1. Shift to where we have a standard CMSIS-PAL
   That is provide a single common NETMF PAL that links against a BSP library for a board that implements the CMSIS-CORE and CMSIS-DRIVER APIs for that device. Thus providing clean isolation between the HAL and PAL and getting NETMF out of defining or controlling the startup sequence, which is a source of a great deal of confusion and frustration when porting to new SOC families.
2. Unify the NETMF PAL with the Llilum managed interop layer APIs
   Unifying the top level APIs allows us to have a common native BSP used for both runtimes. This helps to make migration a lot simpler when you are ready to use Llilum.

The biggest change for NETMF will be at the HAL level, however, by leveraging the existing CMSIS support from silicon vendors the SOC specific parts of that are already there. The major work is in the board specific configuration and drivers for any peripherals on the board that aren't part of the SOC itself. Thus this transition is focused more on what you don't have to write than what you do. In that respect there's not likely much of a migration path for those using the current "do it however you like" model. Rather it would be like starting over but, hopefully, it would be much easier to do since you'd be leveraging the existing library support instead of porting it with lot's of modifications and tweaks to fit into the old NETMF HAL API.

[josesimoes]

I'm not familiar with PAL layer. I've only dip my toes in the HAL. I think I understand what you are suggesting. Can you "draw" a picture of what you are thinking? Maybe a folder structure with the proposed layout to implement such changes... that would help fully understand you ideas on this.

[smaillet-ms]

The best picture available is this one from the Wiki.
The general idea is to Create a PAL that uses CMSIS as the HAL API instead of the current NETMF HAL. This will require some changes to the CLR code that has direct calls to HAL APIs so that the CLR can remain completely dependent on the PAL API only. That will allow existing ports to continue doing what they are doing with minimal change and for new systems to spin up based on the CMSIS libraries.

The folder layout isn't planned to be any different than what we have now. I imagine the new universal CMSIS-PAL could be located DeviceCode\Targets\OS\CMSIS-PAL. This would likely just reference the Targets\OS\CMSIS-RTX support already present.

It might be useful to further adjust the %SPOCLIENT%\CMSIS folder a layout that matches the standard ARM tooling CMSIS-PACK repository. Thus the current folder would be moved to %SPOCLIENT%\Pack\ARM\CMSIS.

Ideally we could use the pack folders already installed by the MDK tools if you have them installed so that we'd have an Environment var to point to the pack root (i.e. CMSIS_PACK_ROOT). Unfortunately, we need to add the NETMF project files into the picture so that we can actually build the code. Thus we either need to completely replicate the pack folder in the NETMF tree (Basically what we are doing now) OR we change our project files to reference the source files form their pack install location. For example, the dotnetmf.proj file for RTX
we'd have something like this:
<Compile Include="%CMSIS_PACK_ROOT%\ARM\CMSIS\4.3.0\CMSIS\RTOS\RTX\SRC\HAL_CM.c" />

Thus the source can remain intact and isolated from the NETMF build requirements. (e.g. reduce the copying and tweaking needed to leverage the existing code for NETMF.)

Further, using the standard layout allows using the standard pack mechanism to download and install packs. While we don't want to ties ourselves to only supporting CMSIS we should be able to support it as it is intended to be used without needing to perform significant "porting" efforts. As mBed 3.0 evolves we can look into the "Yota" system they are using, For managed code and our own code we can use Nuget. With some time and experience with those mechanisms we should be able to formulate an abstraction to enable a NUGET like development experience that incorporates multiple packaging technologies. (But that's a ways out yet - baby steps...) For now I'd be happy with download and install the packs manually. As long as we can use them without modification we've solved the major problem, the rest is about usability and convenience.

[josesimoes]

If the goal is to make NETMF "hardware agnostic" the current DeviceCode should live under each Solution folder, correct?

Having an RTOS "under" the NETMF, or should I say the CLR?, is probably a good idea. I mean always, not only because of the network. This will help better management on power, having async drivers, non blocking tasks, etc.

Should we follow a complete "mBeddish" approach here stick to that architecture? From what I can see on their roadmap for v3 they'll have a lot of good stuff coming that NETMF could benefit from.

[smaillet-ms]

Device code is where it is, if for no other reason than to not break existing users. Additionally it should stay there as the intent for that model was that you would likely have the same peripherals hanging of a several chips (or even on the same SOC within a family) that concept still applies and, for any platforms not using CMSIS we still need to have a model for sharing common code so that's where it goes.

Forget mBed for the moment v3 is still pre-release and v2.0 is a completely different animal from CMSIS and mBed OS 3.0 - CMSIS is real now, stable with LOTS of silicon vendor support. Once we've got CMSIS worked out we can loop back to an mBed PAL or any other "common" layer for other architectures like Zephyr A key point to remember is that NETMF is CPU and software platform neutral.

CLR doesn't require an RTOS underneath and we won't be adding a requirement for one. It does it's own threading, which, due to the nature of the interpreter is much more efficient than a native code kernel. (The memory overhead for storing a thread is quite a bit smaller for the interpreter as it doesn't need to save any CPU registers or state, etc...)

[josesimoes]

I believe that is very important to have all this working with mkd and gcc, so please keep that in mind.

All in all I understand that you have all that pretty much figured out, so I'm not seeing that any further input is required at this time...

[josesimoes]

Regarding the hardware initialization: I'm not sure about the benefits of having the scatter file being parsed/generated from those xml files on the solution folders.
Unless I'm missing something here it seems that they don't add any added value being generated this way. It would seem much simpler, intuitive and easier to understand if they were plain, simple ld files that are edited as needed and passed to the linker.

[smaillet-ms]

They are xml and generate the output file because not all of the tools use ld files, in fact the arm tools use scatter files with a completely different syntax - GCC uses ld scripts. Other tool chains use other solutions. The use of XML helps to abstract all of that and enables tooling to parse them more simply. (e.g. we don't need a unique link/locate language parser for every tool-chain supported, just an XML parser will do). While I think the usefulness of the XML is probably coming to an end, unless/until the entire build system is re-worked, I don't see a lot of value in re-working just that.

[josesimoes]

Understood about the loader files. I was - wrongly - under the impression that the mdk and the gcc ld files were equivalent, just had different name and extension.

[josesimoes]

@smaillet-ms could you please add a new branch to submit PRs for this?

[josesimoes]

@smaillet-ms what should be the naming convention/layout for the "new" DeviceCode folder?

Something like...
DeviceCode\Targets\Native\STM32F0_CMSIS
DeviceCode\Targets\Native\STM32F4_CMSIS

Or...
DeviceCode\Targets\Native\CMSIS\STM32F0
DeviceCode\Targets\Native\CMSIS\STM32F4