Move to standalone libc

[projectgus]

Currently libc functions are provided in libmain. Move to a standalone open source libc.

The new newlib 2.2.0 "nano" features may be a good fit.

Unsure whether it's easier to maintain a separate libc repo and add a binary libc to esp-open-rtos, or build an RTOS-specific libc as yet another source-based "component".

[projectgus]

This looks like an interesting project:
https://github.com/anakod/esp_microc

It doesn't track the upstream git repository, unfortunately. And some of the changes are probably specific "IoT SDK" compatibility. But it's still probably a good approach to follow.

[pfalcon]

It's the same problem as described in #8 (comment) . anakod guy popped out of nowhere, because esp8266 gave him enlightment and sense of life. So he took a generic library, and started to turn it into a product-specific crap. Poor dude didn't know that esp8266 is left to live half-year or year, then it will be displaced by some other product, just the same as it has always been before. But anakod guy doesn't know that, because he popped out of nowhere. But we didn't, we're around long enough to know how it works. So, approach like above can never be good.

[projectgus]

Hi Paul,

I appreciate you feel passionately about the importance of doing things correctly. However please be polite when discussing other people's open source contributions. We don't know their motivations, their requirements, or their backgrounds. If their work isn't suitable for our purposes then that is our problem not theirs.

Despite the reservations I mentioned in the comment I am still glad this project exists and that anakod decided to post it on github, and I'm confident I can learn things from it despite not using it directly.

Angus

[pfalcon]

We don't know their motivations, their requirements, or their backgrounds.

Well in principle, I know, because I was there at my time. I also know specifically for this case, because I read esp8266.ru, where these people communicate. I can assure that their motivation isn't governed by the desire to provide sustainable availability of open-source WiFi system, but by a desire to play with a toy du jour.

Despite the reservations I mentioned in the comment I am still glad this project exists and that anakod decided to post it on github, and I'm confident I can learn things from it despite not using it directly.

Well, it's not enough to be glad and appreciate (what I certainly do). The talk is about whether to reuse their stuff, which in opensource world means participate in their project and spend own time (life) on that - I hope you're not just considering making fork of a fork. Then to dismiss their work instead of participating in it, there should be strong reasons. And I know you're nice guy and won't put it like that, so I happy to utter such strong reasons from outside ;-).

But again, I truly appreciate their work, even if, fairly speaking (let me cut truth again ;-) ), most things to learn from them is how not to do it ;-). I especially appreciate pvvx's efforts - they are not reusable in direct (source code) form, but then it means he talks from the other side of chinese wall, which adds validity to reverse-engineering process. (He came up with his own from-scratch SDK if you noticed: https://github.com/pvvx/SDKnoWiFi . Reason #1 to stay away from it: invalid license. But note that I'm on a cross-road: seeing your and his efforts, I need to decide which to follow, and politically correct indecisiveness of "those both guys are doing awfully great work" doesn't help me or somebody else. Some of you must be doing it "righter", and at the same time any can be making "big mistakes" which should alert other people and save time (of life)).

[pfalcon]

Anyway, let's get this back into productive vein.

But then, how libc problem comes up at all? If you use esp-open-sdk, then it does come with libc - simply because it's normal for an open-source toolchain to come with libc and that's how crosstool-NG (as used by esp-open-sdk) builds it. The actual implementation is the standard for gcc baremetal targets - i.e., newlib.

Caveat: it's not exactly "nano". But it's a known problem reported upstream (to gcc-xtensa maintainer): jcmvbkbc/crosstool-NG#2 . His message: I don't bother with (much of) libc myself, but feel free to do specific suggestions. Note that @nekromant's suggestion of having a libc version which puts itself into SPI flash was contributed by him into esp-open-sdk (you can link with -lcirom).

Not good enough for you? 1) Mind the premature optimization; 2) consider contributing to improve the existing standard solution - it will benefit everyone, not just today's pet project.

There's also other choice - choosing something else. But let's look at the proposed alternative - patched musl. I don't know how that idea came into people's minds. Musl is marketed as library for embedded Linux. That's pretty different from embedded microcontroller-level baremetal. Did somebody really compare newlib and musl? Pretty much unlikely *, so people who chose musl chose it randomly (googled up something, went on patching spree, not caring that there's more suited libc right there before their eyes).

(* I looked for such comparison 6-12 months ago, there was none. I pretty much doubt that people who called a musl fork "esp_microc" did such analysis, and would love to be proven wrong.)

[projectgus]

Note that @nekromant's suggestion of having a libc version which puts itself into SPI flash was contributed by him into esp-open-sdk (you can link with -lcirom)

Thanks for the pointer to this. I'd seen the issue about libc size in IRAM but I'd missed that esp-open-sdk already ran an objcopy to produce an irom version. Very handy.

Newlib is a perfectly good choice, and having it ready to go like this sways my judgement significantly. There's some scope for futher optimisation some time in the future by updating to newlib 2.2.0 which has a few more of the -nano patches, too.

There is one other optimisation I have in mind, to try and reduce the amount of dram used by string literals, but this can probably be done without touching the libc itself.

[pfalcon]

There is one other optimisation I have in mind, to try and reduce the amount of dram used by string literals, but this can probably be done without touching the libc itself.

That's a wild idea I had in mind too - why don't we come to @jcmvbkbc and say: "Hi, why don't you add an attribute to a pointer which would cause any dereference to work on 4-byte alignment, and then sub-field be extracted based on low address bits?" He'd answer: "Hey, but that's not efficient, you probably want to optimize loops which access sequential addresses", but we answer: "Hey, we don't care about your crappy architecture". And probably would add: "Add gcc should be able to optimize such loops itself, no? If no, clang will eat its flesh, soon". Should be easy task for him, don't you think, instead of us writing macros and duplicating each string-taking function as if we were sitting on a goddamn AVR. Wait, we'd still need to duplicate funcs even with that feature. The point, we could skip that, if not caring about each cycle of speed.

[jcmvbkbc]

"why don't you add an attribute to a pointer which would cause any dereference to work on 4-byte alignment, and then sub-field be extracted based on low address bits?"

Because attribute "packed" may be used to the same effect?
Though I'm not sure what strings have to do with it.

[pfalcon]

Because attribute "packed" may be used to the same effect?
Though I'm not sure what strings have to do with it.

Attribute "packed" applied to "char *" will allow that pointer to point to memory-mapped (cached) SPI flash region and be dereferenced without exceptions (because that area is known to require access as aligned 32-bit words)?

[jcmvbkbc]

Ah, ok, I though of a different issue. This one is indeed problematic. But I'm not sure pointer attribute is the best solution here. It doesn't create a new type so you'd need to manually propagate use of that attribute to every function that may deal with data in instruction memory. Providing exception handler looks much easier.

[pfalcon]

It doesn't create a new type so you'd need to manually propagate use of that attribute to every function that may deal with data in instruction memory.

Yes, it's definitely not a panacea, but is a way to quickly write/port existing code. Yes, it will lose on performance side "a lot", but there's pretty big subset of apps where it's not important. As I mentioned above, I imagine 2 usecases for this feature: 1) ~every function in some library accepts pointer arguments with that attribute defined, then user just call functions in that lib with any pointer, which then will be steadily slow; 2) the same trick is used as often used on explicitly Harvard architectures (like AVR mentioned above) - 2 or more versions of function is defined, e.g. printf_ram() & printf_rom(), and user needs to care which to use, but gets optimum performance.

All in all, let me know if that makes any sense to you and you would remotely be interested to implement that, I'll submit proper gcc-xtensa ticket.

Providing exception handler looks much easier.

But that probably will be much slower, and also not portable (i.e. burden to implement it lies on a user vs compiler hacker, and average user would be much less qualified to implement it than average compiler hacker corresponding feature in a compiler ;-) ).

[jcmvbkbc]

But that probably will be much slower, and also not portable

About 2 orders of magnitude slower I'd expect. But it may be implemented as a library that needs one initialization call and no need for user intervention at all.

Also I don't think it's that easy on the compiler side as it may seem, at least adding new attribute definitely involves many parts of the compiler beginning with language front-end.
OTOH if we start with an option that makes all 8-bit accesses compiled that way that seems feasible and only 6 times slower. I'll try that.