Future of .NETMF (take #999999)

[josesimoes]

It’s been 3 months + 1 day since #491 was published. Along with #493, a reasonable amount of discussion was generated and some very interesting points were made. We even got a new branch on the repo for a “v5” where some high level documentation was stored.

Now let me ask you this: what has really changed since then?
Where is .NETMF today compared to where it was 3 months, 6 months, 12 months ago?
Has it moved forward?
Do we have more community engagement?
Were any of the main issues that plague .NETMF solved or at least addressed?
Do we have a plan? A roadmap?
Is there a team working on any of this?

I’ll dare to say that the .NETMF project hasn’t – if we want to be totally honest about it – moved an inch.

It’s about time that we stop beating around the bush and get to the point: .NETMF is not getting anywhere and it won’t go anywhere with what is there today!! This should have become quite obvious by now...

@smaillet-ms, because you are .NETMF project main contact let me ask you this: what else is needed for you to let community people in? What else is needed for you to relinquish total control of .NETMF?

If nothing changes (and I mean drastically and really fast) the moribund .NETMF will be completely dead. (Except on a few repos here and there were it will live forever that belong to people that really love it and will keep working on it no matter what)

Note: this will be my last take on .NETMF future and governance. I’ve made a few attempts to shake people’s minds and have them contribute and step in. If I’m the only one seeing things this way, then I’m the one who is wrong and I have to shut the f#$% up.

[piwi1263]

The thumbs up is for never giving up and try over and over again to put life into this repo

If someone knows him, please tell this guy to release it, please :)
http://www.driversedguru.com/wp-content/gallery/beth-images/parking-brake.jpg

[SytechDesigns]

There are a couple of basic problems!

1. As Steve M has [previously commented - MF is not a MS funded project - he is doing this in his own free time!!
2. Commercial support for MF is dying - GHI have announced the end of support for Gadgeteer, which most likely means they will most likely drop MF support next - meaning another of the main MF OEM board/module manufacturers is out of the game. Chris Walker ( Secrete Labs Netduino) I suspect is no longer manufacturing - We ( Sytech designs) no longer make any OEM MF boards, Mountaineer ( Oberon) can no longer spare resources for this - and a number of 'smaller' guys have stopped as well. The only 'new' board I am aware of is the MikroElectonica one. Where as the target market for most of these guys was hobbyist/Makers, they were also the starting block for commercial developments. Where companies investigating whether MF would be a good fit for their project would evaluate on an OEM board, before developing their own system.
   Where this usually fell over was when they tried an OEM board and liked the dev environment, but then looked at their own custom port - at that point it was usually game over.
   We cannot justify the cost of making OEM or evaluation boards, now that you can get a STM Disco/Nucleo board for no money at all.

Problem has always been the porting process and the steep learning curve etc. This is more critical now with the appearance of lots of low cost embedded modules capable of running Linux/Android.

We are lucky in that we can customise a port for a customer fairly painlessly, and as a result we have completed a number of commercial projects based on MF, and the pain of the port is out weighed by the gains in speed of application development, ease of debugging and overall system reliability.

For MF5 to succeed, the porting process needs to be documented and easier to build - both points have already been acknowledged. It kind of makes a lot of sense to maintain the track it seems to have gone down - focusing on Cortex M base processors - even tho it was supposed to be processor independent.
I think the PAL /Hal layer needs to move over to CMSIS based api making changing from say STM Cortex to NXP an easy task. However, you cant just include the stm hal code, each cmsis api function needs to be just the core code required, like the current device code is based on the stm hal - but much, much smaller.
Build wise we need to move over to a VS based build environment and I also think we need to be able to have an MDK (uvision) based project - for those that use MDK not GCC.
The RTX OS interface is more important than just a way of getting lwip more reliable, we intend to leverage it for adding native comms protocols etc. as well.

However all this is a lot of work - and if we rely on Steve M - then frankly he is never going to have enough free time to do this.

The only way this is going to happen is if some people can commit some time to helping out.
I can make a commitment to work on this,. Mainly I don't want the last 6 years of work I have put in on the MicroFramework to be wasted!
So if we want to see this move ahead we need volunteers and management from Steve M and Lorenzo
So any volunteers?

[piwi1263]

I agree with most, not all of your points. One of the points I disagree with is about GHI not supporting MF anymore. Although this is probably closed source but it is NOT dead, on the contrarary read here for more details ... https://www.ghielectronics.com/community/forum/topic?id=23934&page=1

[josesimoes]

@SytechDesigns you bring up very good points.

To be perfectly clear: I have not – now or ever – complaint on what Steve Maillet has contributed for .NETMF.
Relying or even expecting that he is able to (on its own free time or even working 24/7) design it, marketing it, manage the project, manage the repository and code it would be completely delusional and not understanding (at all!) what’s involved. That’s why I bring this matter once again.

On the matter of evaluation boards for hobbyists or companies working on a proof of concept, I believe that the offer from ST (being currently the only fully supported SoC) is quite abundant, cheap and with all the possible variations (different series, more or less peripherals, networking, low end devices, high end devices, etc). Meaning that it’s probably a waste of resources for any company out there to be offering an evaluation board. The SoC manufacturers already to that very well. So let’s set the focus of .NETMF on the firmware to load on those board (evaluation or OEM).

I agree 200% with you on the porting aspects. Being so “painful” and the lack of documentation is a serious (if not the main) blocking aspect of adoption for new platforms/variants/adaptations to custom or evaluation boards. Once you get it is quite easy actually, but you need to get there…

Adapting the HAL/PAL to CMSIS would be a very good approach on “kind of” standardizing the architecture and making it easy to move between different series and even chip makers. As time permits, me and a colleague of mine, we’ve been working on exactly that approach, along with making the porting and feature management much easier and understandable. Take a look at #434.

On the toolchain (for native code and building a .NETMF image) using VS all the way would be awesome. Maintaining both VS and uVision projects seems complicated and requires an extra effort.
Being able to undergo the full development cycle on VS or VS Code (both are available for free) and using GCC opens the possibility to work on this to anyone.
Going the MDK path leaves a lot of people out.

Regarding volunteering: more than volunteering and/or committing to it there are already a number of us actually and actively contributing with code and work. The more people willing to do it the better, but I don’t think that the lack of contributions or people willing/wanting to contribute is the reason for the non-progress of .NETMF…

[cw2]

@SytechDesigns

So any volunteers?

Volunteers for what exactly? As @josesimoes has mentioned already, there are people willing to commit their resources to various kinds of contributions (issues, code, help etc.), but there is nobody at the other side. I don't blame anybody for turning away after their effort is left without any response for weeks, months or even years.

... we need volunteers and management from Steve M and Lorenzo

I don't question their technical knowledge, but IMVHO it has been clear for quite some time they are not able to manage NETMF project in any reasonable way. I understand delays, change of priorities, personal reasons etc., this happens to everybody - but going completely silent for months without any hint has reached the level of being almost insulting. Especially, considering the status of other projects under .NET Foundation umbrella.

As I see it, NETMF is now at the edge of the cliff, already put a foot forward and looking back at @smaillet-ms - unfortunately, there is not much time left...

[smaillet-ms]

TL;DR

After writing this all it came out a bit long so I'll summarize. The success or failure of NETMF (the current interpreter as well as the general concept of .NET for micros) depends entirely on the people who choose to work on it. There is a ton of work that can be done, most of it isn't writing code for an STM32 and may not be of interest or within the capabilities of everyone. The success of NETMF doesn't depend on me, Lorenzo, our employer or any other entity to "manage it". If NETMF is to survive it will do so because enough people are willing, able, and step up to make it useful enough to a sufficiently large enough group of people.

---

there are people willing to commit their resources to various kinds of contributions (issues, code, help etc.), but there is nobody at the other side

What other side are you talking about? Where are these people? Where are their design proposals for any of the open issues? Why haven't any of them bothered to fix the commits that have already been allowed in and broke the GGC builds? (#508) I'm not trying to start a Grand Urinary Olympics here but this isn't so one sided as you seem to think it is. There are a number of pending PRs that I haven't merged because they don't address the busted build. Let's get that fixed then we can bring in the other pieces. There's plenty of work to be done if people want to do it they need to step up. If there is an existing issue filed then propose a solution, discuss or comment on proposals, then implement a PR, engage in reviews of PRs. If there isn't an issue filed, then submit one to define the problem. If everyone waits for someone else to assign tasks to them things will just stall. If there was a huge team with financial interests in the project maybe we could afford someone to manage tasks at that level, but whether we like it or not - that's not the reality here.

A major tenant of Agile Development is the idea of self organizing teams, rather then strict rigid hierarchies. I think NETMF needs that, but it takes a certain kind of commitment and discipline that's quite different from what most people are used to in their professional employment.

---

Now, to the future of NETMF. Some are painting a lot of doom and gloom and bleak dark pictures on NETMF - they are entitled to their opinions. They may even be correct in the end. If you don't want such views to be reality then you need to ask yourself - what are you willing to do to prevent it? If you only need some functionality to continue shipping your own commercial products and don't care if such changes are relevant to anyone else (or even are counter productive to them) then by all means fork the code and have at it. If on the other hand you want/need it to be a platform in and of itself with a vibrant and active user base then ask yourself what is it that prevents that and what are you willing and able to do to get it there.

I for one would like to see NETMF as a platform in and of itself. Though I need to make it clear that doesn't necessarily mean the current code base. More specifically I mean the concept of .NET and especially C# as a language/platform for programming micro controllers in embedded systems. After I posted a set of ideas and a branch for vNext there was a lot of criticism about spending any time on NETMF instead of Llilum (e.g. why not let the interpreter "out to pasture" and push on with AOT compilation with Llilium). Thus, it wasn't clear to me that what I was proposing was really where things should go. While it is work I'd always wanted to see done for NETMF I'm not really interested in doing it as an academic exercise for myself. (I've got other projects for that)

In addition to the pushback at the same time, the basic technologies to build off of weren't quite publicly available or ready for use. This includes the new VS "15" project system support, which as of the latest preview is now pretty stable to build off of. The other core piece for the foundation was CMSIS, since we are looking to re-define the entire HAL/PAL concept for the platform (AOT , interpreted or both, doesn't really matter) there wasn't a lot of value doing that based on the V4.x flavors of CMSIS when V5 was making some significant and valuable changes (especially in the CMSIS-RTOS2 API). Unfortunately, that has been delayed and pushed back several times, and only reached final release state a few days ago.

Since not all of the underpinning tech was ready I've been spending time looking at the AOT side of things. One of the biggest concerns with just shifting all efforts to Llilium is that it is a very complex code base where it would be very hard to ramp up on. Other concerns include that it is very slow to compile and the overall footprint isn't as low as most of us would like/hope for. So I took everything I learned for using LLVM.NET in Llium and the early test project I had created before we moved to build Llilum out of an older project. I'm not claiming that doing Llilum based on the older Zelig code was a wrong or even bad choice - as it gave us something that mostly works and shows there is value to be had in the area of AOT compilation of .NET for micros. However it isn't something that's really a long term solution either. It was designed based on an original idea that everything could be done in C# without any need to interop with existing C code. 10 years ago when it was originally designed that was a plausible, albeit quite bold, story - today - not a chance. So, while we managed to bolt LLVM onto the bottom to replace the old ARMv4 code generation there was a lot where the original Zelig tools and LLVM tool essentially get in each others way.

Given all of that, and my own desire to see a viable AOT solution. I dusted off some early prototyping code with a deliberately naïve plan - do as little as possible in the transform tool and let LLVM do as much as possible. The results of my experiments in this direction are actually pretty surprising. I generated a simple blinky app with no objects/GC (only static data) and an assembly generated from a CMSIS-SVD file to provide direct type safe access to hardware registers. Conversion of the main app, a secondary assembly implementing the equivalent to the mbed HAL GPIO API and the assembly generated from the SVD into a single fully optimized object file takes a little under 8 seconds, and when linked to the standard startup code from CMSIS the entire app comes in a little under 3K. Granted that is without any GC or allocated classes with vtables to take up space. But it shows we can be pretty close to C++ without a GC. The Boehm GC currently in Llilum wouldn't add a huge amount on top of that. Furthermore, the approach I've taken is to generate LLVM bit code modules for each assembly individually, and save them along side the original assembly. Thus a minor change in one assembly means generating only the bit code for that one assembly and all the others are re-used as is. Thus on a minor tweak to the main exe the time to generate the obj file a second time comes down to under 2 seconds

The type safe hardware access portion of that relies on a new C#7 feature - byref returns, while the capability for that has always been in the .NET runtime (C++/CLI uses it) C# has never exposed it until now. This feature along with a special interface and some attributes known to the code generator allows generation of type safe hardware access that ultimately resolves to dereferencing constant pointers as you would get in C.

example:

    /// <summary>Demonstrates a pure C# implementation of the equivalent mBed HAL APIs</summary>
    /// <remarks>
    /// DeviceMap is a static type in an assembly generated from the CMSIS-SVD description
    /// of the on-chip hardware register space. It provides high performance type safe direct
    /// access to the hardware registers. (Code size and perf is equivalent to C code as the
    /// registers are mostly accessed via constant pointers and property methods aggressively inlined).
    /// </remarks>
    public static class GpioApi
    {
        /// <summary>Configures a GPIO pin as an input</summary>
        /// <param name="pinData">GPIO pin definition data structure to hold initialized data for the pin</param>
        /// <param name="pin">Pin id</param>
        /// <param name="mode">Mode for the pin</param>
        public static void ConfigureAsInput( ref PinInfo pinData, PinName pin, PinMode mode = PinMode.PullDefault )
        {
            pinData.Port = ( ( int )pin ) / 32;
            pinData.Mask = 1u << ( ( int )pin ) % 32;
            DeviceMap.GPIO.PORT[pinData.Port].DIR.Value &= ~pinData.Mask;
        }

        /// <summary>Configures a GPIO pin ans an input</summary>
        /// <param name="pinData">GPIO pin definition data structure to hold initialied data for the pin</param>
        /// <param name="pin">Pin id</param>
        /// <param name="value">Initial state of the pin (default == 0)</param>
        public static void ConfigureAsOutput( ref PinInfo pinData, PinName pin, int value = 0 )
        {
            pinData.Port = ( ( int )pin ) / 32;
            pinData.Mask = 1u << ( ( int )pin ) % 32;
            DeviceMap.GPIO.PORT[ pinData.Port ].DIR.Value |= pinData.Mask;
            Write( ref pinData, value );
        }

        /// <summary>Writes a new state for a GPIO pin</summary>
        /// <param name="pinData">Data structure identifying the pin</param>
        /// <param name="value">new value for the pin</param>
        public static void Write( ref PinInfo pinData, int value )
        {
            if (value == 0)
                DeviceMap.GPIO.PORT[pinData.Port].CLR.Value = pinData.Mask;
            else
                DeviceMap.GPIO.PORT[pinData.Port].SET.Value = pinData.Mask;
        }

sample main:

using NXP.LPC1768;

namespace Blinky
{
    /// <summary>Simple test app to validate behavior of code generation</summary>
    public static class Program
    {
        // GPIO pin info for the LEDs on the dev board
        static PinInfo Led1;
        static PinInfo Led2;
        static PinInfo Led3;
        static PinInfo Led4;
        static PinInfo LedRed;
        static PinInfo LedGreen;
        static PinInfo LedBlue;

        const int WaitTimeoutMs = 75;

        /// <summary>Main entry point for the app</summary>
        /// <remarks>
        /// The runtime startup code will call this after initializing. Technically,
        /// the ExportAsAttribute here is redundant as the codegen tool can infer
        /// the name from the assemblies EntryPoint as "main" and effectively apply
        /// the attribute dynamically.
        /// </remarks>
        //[ExportAs( "main")]
        public static void Main( )
        {
            GpioApi.ConfigureAsOutput( ref Led1, PinName.LED1, 1 );
            GpioApi.ConfigureAsOutput( ref Led2, PinName.LED2, 1 );
            GpioApi.ConfigureAsOutput( ref Led3, PinName.LED3, 1 );
            GpioApi.ConfigureAsOutput( ref Led4, PinName.LED4, 1 );
            GpioApi.ConfigureAsOutput( ref LedRed, PinName.p23, 1 );
            GpioApi.ConfigureAsOutput( ref LedGreen, PinName.p24, 1 );
            GpioApi.ConfigureAsOutput( ref LedBlue, PinName.p25, 1 );

            var state = true;
            while( true )
            {
                var activeHigh = state ? 1 : 0;
                var activeLow = state ? 0 : 1;
                GpioApi.Write( ref Led1, activeHigh );
                WaitApi.Wait( WaitTimeoutMs );
                GpioApi.Write( ref Led2, activeHigh );
                WaitApi.Wait( WaitTimeoutMs );
                GpioApi.Write( ref Led3, activeHigh );
                WaitApi.Wait( WaitTimeoutMs );
                GpioApi.Write( ref Led4, activeHigh );
                WaitApi.Wait( WaitTimeoutMs );
                // RGB leds are active low
                GpioApi.Write( ref LedRed, activeLow );
                WaitApi.Wait( WaitTimeoutMs );
                GpioApi.Write( ref LedGreen, activeLow );
                WaitApi.Wait( WaitTimeoutMs );
                GpioApi.Write( ref LedBlue, activeLow ); 
                WaitApi.Wait( WaitTimeoutMs );
                state = !state;
            }
            // Never returns...
        }
    }
}

I'm hoping I can get the basic structure of the experiment cleaned up to a bit more useful state (and maybe even some docs!) by the end of the holiday season.

There's still a long way to go with this. Some of which can be adapted from Llilium and or the CoreCLR work. The HAL layer and interop still needs to be defined, which was the biggest focus of my proposals for NETMF v5, not to mention the problem of the "soft dependencies". Although, in a surprise addition to CMSIS-5 there is generalized "pseudo-project" file that lists the files and headers in a source package but also lists dependencies on packages and interfaces that could serve well in that area.

Whether this has any value to anyone or not is another story, How much functionality is required before it can be considered useful is an open question.

[cw2]

There are a number of pending PRs that I haven't merged because they don't address the busted build.

Perhaps it would help to let the contributors know the reason why the PRs are getting stalled. I guess a simple comment/message "No PRs until #508 is fixed" will be sufficient and appreciated.

Regarding the NETMF v5 - does it mean it will be AOT?

Also, are you aware of CoreRT project?

[smaillet-ms]

I've already made such a statement once but I can add it to all of the ones that have come on since then.

Regarding the NETMF v5 - does it mean it will be AOT?

Maybe, maybe not - depends on the people who are willing and able to step up really. Ultimately there are a few questions there:

1. Is there enough desire for an interpreter and the features it offers that wouldn't be available in an AOT variant
2. If there is value in an interpreter does it need to internally look anything like the current NETMF does? (i.e. could an interpreter be written entirely in C# and compiled to native using the AOT tools, potentially leveraging the same underlying garbage collector/heap?)
3. If yes to any of the above do we have enough people to pull off both?
4. If both have value but we don't have enough resources, which one is more important?

These aren't new questions but as of yet don't have a clear answer. There's been a lot of interest in an AOT solution but concerns over the time to a minimally viable product have people a bit on edge, leading to talk about tweaking the current NETMF a bit in the interim.

Also, are you aware of CoreRT project?

I am aware of the CoreRT project, in the past we've actually had a few conversations with the team driving that work. Unfortunately , the goals and requirements for CoreRT don't align well with those of a micro controller focused solution. While there is likely some things we can learn/leverage from there, it isn't useable as-is and trying to make it meet our needs has a high probability of getting in the way of CoreRT meeting its own goals. One of the bigger challenges CoreRT needs to deal with doesn't exist for the small micro space. (e.g. CoreRT has to contend with dynamic loading of modules and that those modules might have open generic types that other modules need to instantiate. ) On the flip side for the NETMF AOT space we have extreme memory constraints to contend with that CoreRT doesn't. Common concepts (and possibly code) like how types are laid out, how vtables are defined and used, etc... may also be useful to share. Ultimately, all such sharing in either direction needs to consider the impact to the first order goals of each project.

[godefroi]

My votes and opinions count for very little, but here are some of the priorities I see:

• The project should target (or be easy to "port" to) commonly-available and inexpensive microcontrollers. This likely means MCUs that are available as STM Discovery boards and MBed boards.
• The system should allow low-level hardware interfacing with sufficient performance that strong commercial support for "modules" (i.e. sensors, motor drivers, etc) is not required as part of the underlying native code.
• The project should be buildable by members of the community without access to expensive development environments (Kiel, MDK).

Personally, I would give up threading, the GC, and reflection in a heartbeat to have a high-performance close-to-the-metal AOT NETMF. This may be a controversial position, but it is mine.

[miloush]

@godefroi sounds like Llilum to me

[Stabilitronas]

Again, Steve claims that "it's a lot more than you see", but perception is reality. And what I perceive is maybe one small commit a week — for more than half a year now. If there really is something "a lot more", then maybe it's just about time to show it?

Steve still dreams of a very abstract and cool platform, that works on everything and has everything, but, judging on the opinions on this thread and in #491, it doesn't look like he has lots of supporters believing he can actually do it. V5 spec was a short spark of hope... that ended in 22 commits over 4 months.

Now that the last big commercial passenger (==GHI) left the train, I see no reason why @josesimoes couldn't simply fork NETMF and go his own route, trying to bring in folks that don't like the way NETMF is currently going along. If he said "alright, let's just ditch MDK and KEIL support, GC and other 80% of code, let's just make it work, work fast, compile in 1 minute, and run on a 8$ Nucleo board out-of-the-box", that would certainly attract my attention. Surely, I'm not the center of the world, but that's my opinion anyway.

[godefroi]

@miloush While Llilum is interesting, Steve's proof-of-concept there is much more elegant.

[smaillet-ms]

@Stabilitronas says:

Again, Steve claims that "it's a lot more than you see",

I am making no such claims - The only thing I've said that I can think of that might have been interpreted that way was the point that the issues being discussed aren't as one sided as some claimed.

let's just ditch MDK and KEIL support, GC and other 80% of code, let's just make it work, work fast, compile in 1 minute, and run on a 8$ Nucleo board out-of-the-box", that would certainly attract my attention

While the "Ditch" part of that is an open question here, the rest is essentially what I am saying I have a good basis for in the AOT work I've been doing. It isn't quite up to where its useful to publish (There's still some significant refactoring churn to complete) but I am hoping to get that done by the end of the holidays. It doesn't support all the base instructions and pretty much none of the object model ones yet, but one I get the refactoring done that is work that could be done by others. (A few major architectural choices regarding types and the GC need to be defined, but again multiple people can participate there especially in reviewing what CoreRT, Llilum, LLilc and any other similar platforms do to weigh the various tradeoffs etc...) Llilium has more features and functionality by a large factor than I currently do and may be a way to get some projects started despite its issues with the compilation time (and memory usage during compilation).

I can't, and wouldn't if I could, dictate what happens next for the current NETMF code base. If there are enough people who see value in making the effort to V5 or some other proposed design - great let's go. Otherwise I'm going to continue my focus on the AOT tooling. If others want to join me on that, great.