generate errno definitions from CSV #15579

yamt · 2025-01-16T05:41:59Z

Summary

generate errno definitions from a CSV file for flexibility.

i plan to use this CSV file for other things like host <-> nuttx errno conversion logic.

Impact

Testing

i investigated the diff between the generated file and the original errno.h
bulid-tested sim:toywasm

nuttxpr · 2025-01-16T05:47:32Z

[Experimental Bot, please feedback here]

Fill In The Commit Message: This PR contains a Commit with an Empty Commit Message. Please fill in the Commit Message with the PR Summary.

This PR does not fully meet the NuttX requirements. While it provides a summary of what is being changed, it lacks crucial details.

Here's what's missing:

Summary:
- Why is this change necessary? What problem does it solve? Is the current errno handling inflexible? How?
- How exactly does the generation work? Is there a script? Where does the CSV file live? How is it integrated into the build system?
- Issue references are missing. Even if there isn't a directly related issue, it's good practice to create one to discuss the proposed change before submitting a PR.
Impact: This section is completely empty. All points need to be addressed, even if the answer is "NO". Justify the "NO" answers. For example: "Impact on user: NO. This is an internal change to errno generation and does not affect the API." If the answer is "YES," explain the impact in detail. For instance, will existing user code break? Will new configuration options be added?
Testing:
- No build host or target information is provided. What platforms was this tested on? Be specific (e.g., "Linux, x86_64, GCC 12.1", "sim:qemu-x86_64").
- No test logs are provided. Demonstrate that the change works as intended. Show relevant output before and after the change. Even a simple "printf" demonstrating the generated errno values would be helpful.

The PR needs to provide significantly more detail to be considered complete. Addressing the above points will make it easier for reviewers to understand and evaluate the change.

yamt · 2025-01-16T06:32:28Z

i have a question: is python acceptable for this kind of tools these days?
honestly speaking, i'm not inclined to write this in C.

raiden00pl · 2025-01-16T08:07:59Z

i have a question: is python acceptable for this kind of tools these days?
honestly speaking, i'm not inclined to write this in C.

I don't think so, or I missed something. If the build system is to depend on python, then this is a more serious issue to discuss in the community. Or was it introduced quietly and no one noticed?

EDIT: if python is used to generate the file, but not when build process, then it's probably OK?

yamt · 2025-01-16T08:26:18Z

i have a question: is python acceptable for this kind of tools these days?
honestly speaking, i'm not inclined to write this in C.

I don't think so, or I missed something. If the build system is to depend on python, then this is a more serious issue to discuss in the community. Or was it introduced quietly and no one noticed?

EDIT: if python is used to generate the file, but not when build process, then it's probably OK?

the script this PR added (mkerrno.py) is not meant to be used during a build.
i expect a developer who modified errno.csv to run it to regenerate some source files.

otoh, some of existing python scripts in the tools dir seem to be used during a build.
eg. mkallsyms.py, kasan_global.py

raiden00pl · 2025-01-16T08:49:07Z

the script this PR added (mkerrno.py) is not meant to be used during a build.
i expect a developer who modified errno.csv to run it to regenerate some source files.

then it's definitely ok.

otoh, some of existing python scripts in the tools dir seem to be used during a build.
eg. mkallsyms.py, kasan_global.py

yeah, I just saw this. So it was introduced without much discussion and even requirements in the documentation hasn't been updated. I see it's only for the case when KASAN is enabled, but this doesn't change the fact that python is now required to build....

xiaoxiang781216 · 2025-01-17T15:31:13Z

@yamt what's the benefit to generate the errno from a csv, but not modify errno.h direclty?

yamt · 2025-01-18T05:51:54Z

@yamt what's the benefit to generate the errno from a csv, but not modify errno.h direclty?

i want to generate other things as well, which are difficult or impossible with C preprocessor.
eg.

#if defined(EPERM)
case EPERM: return NUTTX_EPERM;
#endif

xiaoxiang781216 · 2025-01-19T09:20:08Z

@yamt what's the benefit to generate the errno from a csv, but not modify errno.h direclty?

i want to generate other things as well, which are difficult or impossible with C preprocessor. eg.
#if defined(EPERM)
case EPERM: return NUTTX_EPERM;
#endif

I still can't fully understand what you want to achieve with this code snippet, could you explain more? In most case, it's be better to complish the generation work as macro as we can.

xiaoxiang781216 · 2025-01-19T09:27:12Z

the script this PR added (mkerrno.py) is not meant to be used during a build.
i expect a developer who modified errno.csv to run it to regenerate some source files.

then it's definitely ok.

but why not generate automatically?

otoh, some of existing python scripts in the tools dir seem to be used during a build.
eg. mkallsyms.py, kasan_global.py

yeah, I just saw this. So it was introduced without much discussion and even requirements in the documentation hasn't been updated. I see it's only for the case when KASAN is enabled, but this doesn't change the fact that python is now required to build....

both(kasan.py and mkallsyms.py) are advanced features, which are disabled by default. so python isn't required before you enable these from defconfig.

But, I need mention that shell script(include the tools invoked in it) is hard to work in Windows native environment. If we want to improve the Windows native develop experience, python is the best script we can choice.

yamt · 2025-01-19T10:00:58Z

@yamt what's the benefit to generate the errno from a csv, but not modify errno.h direclty?

i want to generate other things as well, which are difficult or impossible with C preprocessor. eg.
#if defined(EPERM)
case EPERM: return NUTTX_EPERM;
#endif
I still can't fully understand what you want to achieve with this code snippet, could you explain more?

convert host errno to nuttx errno.

In most case, it's be better to complish the generation work as macro as we can.

i can't think of any way to do the equivalent with C preprocessor alone.

xiaoxiang781216 · 2025-01-19T15:09:45Z

@yamt what's the benefit to generate the errno from a csv, but not modify errno.h direclty?

i want to generate other things as well, which are difficult or impossible with C preprocessor. eg.
#if defined(EPERM)
case EPERM: return NUTTX_EPERM;
#endif
I still can't fully understand what you want to achieve with this code snippet, could you explain more?
convert host errno to nuttx errno.

why not simply define EINVAL_LINUX, EINVAL_MACOS, EINVAL_WINDOWS directly, instead through errno_xxx.csv and mkerrno.py.

In most case, it's be better to complish the generation work as macro as we can.

i can't think of any way to do the equivalent with C preprocessor alone.

sorry, I don't see any hard block issue to implement the conversion in c entirely.

yamt · 2025-01-20T02:40:49Z

@yamt what's the benefit to generate the errno from a csv, but not modify errno.h direclty?

i want to generate other things as well, which are difficult or impossible with C preprocessor. eg.
#if defined(EPERM)
case EPERM: return NUTTX_EPERM;
#endif
I still can't fully understand what you want to achieve with this code snippet, could you explain more?
convert host errno to nuttx errno.
why not simply define EINVAL_LINUX, EINVAL_MACOS, EINVAL_WINDOWS directly, instead through errno_xxx.csv and mkerrno.py.

EPREM in the above snippet is the host one. (just use the host errno.h)
i don't see any benefit to have separate definitions for each platforms.

In most case, it's be better to complish the generation work as macro as we can.

i can't think of any way to do the equivalent with C preprocessor alone.

sorry, I don't see any hard block issue to implement the conversion in c entirely.

well, of course, i can keep the csv and script private and only submit the generated C files to nuttx.
i don't see any benefit to hide how these files were actually generated though.

yamt · 2025-01-20T03:20:21Z

the script this PR added (mkerrno.py) is not meant to be used during a build.
i expect a developer who modified errno.csv to run it to regenerate some source files.

then it's definitely ok.

but why not generate automatically?

because

i expect that it's very rare to change the errno definitions
i expect people are not happy with python requirement for just a build
i don't want to write this in C
it's cumbersome to update multiple build frameworks (make and cmake)

xiaoxiang781216 · 2025-01-20T03:59:11Z

@yamt what's the benefit to generate the errno from a csv, but not modify errno.h direclty?

i want to generate other things as well, which are difficult or impossible with C preprocessor. eg.
#if defined(EPERM)
case EPERM: return NUTTX_EPERM;
#endif
I still can't fully understand what you want to achieve with this code snippet, could you explain more?
convert host errno to nuttx errno.
why not simply define EINVAL_LINUX, EINVAL_MACOS, EINVAL_WINDOWS directly, instead through errno_xxx.csv and mkerrno.py.
EPREM in the above snippet is the host one. (just use the host errno.h) i don't see any benefit to have separate definitions for each platforms.

In most case, it's be better to complish the generation work as macro as we can.

i can't think of any way to do the equivalent with C preprocessor alone.

sorry, I don't see any hard block issue to implement the conversion in c entirely.

well, of course, i can keep the csv and script private and only submit the generated C files to nuttx. i don't see any benefit to hide how these files were actually generated though.

Does your solution share one errno.csv for all OS or have the seperate errno.csv for each OS? The complex of include/errno.csv and include/errno.h is almost same, so whether we can share the same errno.csv is a key point.

yamt · 2025-01-20T04:28:44Z

@yamt what's the benefit to generate the errno from a csv, but not modify errno.h direclty?

i want to generate other things as well, which are difficult or impossible with C preprocessor. eg.
#if defined(EPERM)
case EPERM: return NUTTX_EPERM;
#endif
I still can't fully understand what you want to achieve with this code snippet, could you explain more?
convert host errno to nuttx errno.
why not simply define EINVAL_LINUX, EINVAL_MACOS, EINVAL_WINDOWS directly, instead through errno_xxx.csv and mkerrno.py.
EPREM in the above snippet is the host one. (just use the host errno.h) i don't see any benefit to have separate definitions for each platforms.

In most case, it's be better to complish the generation work as macro as we can.

i can't think of any way to do the equivalent with C preprocessor alone.

sorry, I don't see any hard block issue to implement the conversion in c entirely.

well, of course, i can keep the csv and script private and only submit the generated C files to nuttx. i don't see any benefit to hide how these files were actually generated though.
Does your solution share one errno.csv for all OS or have the seperate errno.csv for each OS? The complex of include/errno.csv and include/errno.h is almost same, so whether we can share the same errno.csv is a key point.

errno.csv only defines things for nuttx.

i have no plan to have csv files for other OSes. (linux, macos, etc)
just because i don't see any needs for my purposes.

xiaoxiang781216 · 2025-01-20T05:27:38Z

Does your solution share one errno.csv for all OS or have the seperate errno.csv for each OS? The complex of include/errno.csv and include/errno.h is almost same, so whether we can share the same errno.csv is a key point.

errno.csv only defines things for nuttx.

So, why do we introduce errno.csv? It's more simple to modify errno.h directly to add/remove/change errno.

i have no plan to have csv files for other OSes. (linux, macos, etc) just because i don't see any needs for my purposes.

Sorry, I still can't understand how csv could help to fix hostfs issue. With the current patchset, I don't see any benefit to generate errno.h from mkerrno.py and errno.csv.

yamt · 2025-01-20T05:35:45Z

Does your solution share one errno.csv for all OS or have the seperate errno.csv for each OS? The complex of include/errno.csv and include/errno.h is almost same, so whether we can share the same errno.csv is a key point.

errno.csv only defines things for nuttx.

So, why do we introduce errno.csv? It's more simple to modify errno.h directly to add/remove/change errno.

because, as explained in the commit message, i want to generate more than errno.h (and automatically keep them consistent)

i have no plan to have csv files for other OSes. (linux, macos, etc) just because i don't see any needs for my purposes.

Sorry, I still can't understand how csv could help to fix hostfs issue. With the current patchset, I don't see any benefit to generate errno.h from mkerrno.py and errno.csv.

there is no benefit from this PR alone because it only generates errno.h at this point.

xiaoxiang781216 · 2025-01-20T05:54:01Z

could you provide other patch? so I can understand the whole picture.

yamt · 2025-01-20T05:59:01Z

could you provide other patch? so I can understand the whole picture.

ok.

(it might take a bit long to find time to work on this.)

yamt · 2025-04-21T00:51:22Z

could you provide other patch? so I can understand the whole picture.

ok.

(it might take a bit long to find time to work on this.)

done.

i just added changes for sim hostfs to this PR.
if you prefer to make it a separate PR, please tell me.

yamt · 2025-04-21T00:53:40Z

hm, the latest nxstyle seems more nervous than the binary i happened to have. let me fix.

this is a machine-friendly version of errno.h using this file, i plan to generate: * the corresponding C definitions in errno.h. * strerror tables. * host <-> nuttx errno conversion logic for sim. Signed-off-by: YAMAMOTO Takashi <[email protected]>

Signed-off-by: YAMAMOTO Takashi <[email protected]>

yamt · 2025-04-21T01:36:17Z

hm, the latest nxstyle seems more nervous than the binary i happened to have. let me fix.

done

xiaoxiang781216 · 2025-04-21T03:14:02Z

include/errno_defs.h

@@ -0,0 +1,311 @@
+/****************************************************************************
+ * include/errno_defs.h


it's better to generate errno_defs.h automatically instead adding to git, to simplify the process of errno modification.

it would be very rare to modify the definitions.
IMO, it doesn't warrant the effort to write C programs to avoid build-time python dependencies.

xiaoxiang781216 · 2025-04-21T03:19:24Z

tools/mkerrno.py

@@ -0,0 +1,77 @@
+#!/usr/bin/env python3
+############################################################################
+# tools/mkerrno.py


we can achieve the similar result by c macro instead python script, please check how syscall is implemented:
https://github.com/apache/nuttx/blob/master/include/sys/syscall.h
https://github.com/apache/nuttx/blob/master/include/sys/syscall_lookup.h
https://github.com/apache/nuttx/blob/master/syscall/syscall_wraps.h
https://github.com/apache/nuttx/blob/master/syscall/syscall_names.c
https://github.com/apache/nuttx/blob/master/syscall/syscall_stublookup.c

sorry, can you explain how you can generate an equivalent of C code in this PR,
eg.

#if defined(EPERM) case EPERM: return 1; #endif

with the approach?

i suppose you mean to have

ERRNO_ITEM(EPERM, 1, "Operation not permitted")

but i'm not sure how you can define ERRNO_ITEM which expands to an equivalent of the above mentioned snippet.

also, the syscall stuff actually has another list: syscall.csv. having multiple lists which we need to maintain the consistency is what i want to avoid.

sorry, can you explain how you can generate an equivalent of C code in this PR, eg.

#if defined(EPERM) case EPERM: return 1; #endif

with the approach?

it's hard to add #if defined in macro definition, but could doable with the macro trick like this:
https://stackoverflow.com/questions/72266480/can-ifdef-be-used-inside-a-macro

i suppose you mean to have

ERRNO_ITEM(EPERM, 1, "Operation not permitted")

but i'm not sure how you can define ERRNO_ITEM which expands to an equivalent of the above mentioned snippet.

also, the syscall stuff actually has another list: syscall.csv. having multiple lists which we need to maintain the consistency is what i want to avoid.

syscall.csv define the function prototype, but errno doesn't need it, so the header file is enough.

sorry, can you explain how you can generate an equivalent of C code in this PR, eg.

#if defined(EPERM) case EPERM: return 1; #endif

with the approach?

it's hard to add #if defined in macro definition, but could doable with the macro trick like this: https://stackoverflow.com/questions/72266480/can-ifdef-be-used-inside-a-macro

well, it might work for a single macro like DEBUG.
however, in this case, we have to test every Exxx macros.

we can use ERRNO_ITEM to iterate each EXXX and generate individual macro.

we can use ERRNO_ITEM to iterate each EXXX and generate individual macro.

how?
i couldn't think of any reasonable implementations.

I will find some time in the weekend to try the pure macro solution.

thank you.

here's what i managed to write so far: https://github.com/yamt/garbage/tree/master/c/cpp1
i feel it isn't a reasonable implementation though.

yamt · 2025-04-21T04:34:14Z

cmake build is failing because it requires the host sources under posix or win.
while i guess this file works for win as well, i probably will move it under posix because it's the only user right now. (and i'm not in a mood to work with cmake right now.)

this has two purposes: * reduce linux assumptions in sim (this is a follow-up of apache#15552) * demonstrate errno.csv can be used for other purposes Signed-off-by: YAMAMOTO Takashi <[email protected]>

this is a follow-up of apache#15552 Signed-off-by: YAMAMOTO Takashi <[email protected]>

tested with ELOOP, which is 62 on macOS and 40 on NuttX, on sim/macOS. ```shell spacetanuki% ln -s a a spacetanuki% cat a cat: a: Too many levels of symbolic links spacetanuki% ./nuttx NuttShell (NSH) NuttX-10.4.0 nsh> mount -t hostfs -o fs=. /mnt nsh> cat /mnt/a nsh: cat: open failed: 40 nsh> ``` this is a follow-up of apache#15552 this fixes a non-linux regression in apache#15535 Signed-off-by: YAMAMOTO Takashi <[email protected]>

yamt · 2025-04-21T10:13:19Z

cmake build is failing because it requires the host sources under posix or win. while i guess this file works for win as well, i probably will move it under posix because it's the only user right now. (and i'm not in a mood to work with cmake right now.)

done

github-actions bot added Area: Tooling Area: OS Components OS Components issues Size: L The size of the change in this PR is large labels Jan 16, 2025

yamt force-pushed the errno branch from 1c2abab to a52c0dd Compare January 16, 2025 05:45

yamt mentioned this pull request Jan 16, 2025

hostfs_open:return errno from host #15535

Merged

lupyuen approved these changes Jan 16, 2025

View reviewed changes

simbit18 approved these changes Jan 17, 2025

View reviewed changes

yamt force-pushed the errno branch from a52c0dd to 98a3e06 Compare April 21, 2025 00:50

github-actions bot added Arch: simulator Issues related to the SIMulator Size: XL The size of the change in this PR is very large. Consider breaking down the PR into smaller pieces. and removed Size: L The size of the change in this PR is large labels Apr 21, 2025

yamt added 4 commits April 21, 2025 10:00

add errno.csv

2bf5af3

this is a machine-friendly version of errno.h using this file, i plan to generate: * the corresponding C definitions in errno.h. * strerror tables. * host <-> nuttx errno conversion logic for sim. Signed-off-by: YAMAMOTO Takashi <[email protected]>

add a script to generate errno definitions for errno.h

c3119f3

Signed-off-by: YAMAMOTO Takashi <[email protected]>

generate include/errno_defs.h

1820796

Signed-off-by: YAMAMOTO Takashi <[email protected]>

include/errno.h: consume errno_defs.h

ca5e3db

Signed-off-by: YAMAMOTO Takashi <[email protected]>

yamt force-pushed the errno branch 3 times, most recently from a0e9537 to 38246e7 Compare April 21, 2025 01:33

xiaoxiang781216 reviewed Apr 21, 2025

View reviewed changes

yamt force-pushed the errno branch from 38246e7 to c0ab573 Compare April 21, 2025 03:49

yamt force-pushed the errno branch from c0ab573 to 251505e Compare April 21, 2025 09:39

yamt added 4 commits April 21, 2025 18:45

add a script to generate errno conversion function for sim

10b9a71

this has two purposes: * reduce linux assumptions in sim (this is a follow-up of apache#15552) * demonstrate errno.csv can be used for other purposes Signed-off-by: YAMAMOTO Takashi <[email protected]>

generate sim_hosterrno.c

3de33fc

this is a follow-up of apache#15552 Signed-off-by: YAMAMOTO Takashi <[email protected]>

sim: add host_errno_to_nx

3e46fd5

this is a follow-up of apache#15552 Signed-off-by: YAMAMOTO Takashi <[email protected]>

yamt force-pushed the errno branch from 251505e to 8640ba1 Compare April 21, 2025 09:46

jerpelea approved these changes Apr 22, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

generate errno definitions from CSV #15579

generate errno definitions from CSV #15579

yamt commented Jan 16, 2025 •

edited

Loading

nuttxpr commented Jan 16, 2025

yamt commented Jan 16, 2025

raiden00pl commented Jan 16, 2025 •

edited

Loading

yamt commented Jan 16, 2025

raiden00pl commented Jan 16, 2025

xiaoxiang781216 commented Jan 17, 2025

yamt commented Jan 18, 2025

xiaoxiang781216 commented Jan 19, 2025

xiaoxiang781216 commented Jan 19, 2025

yamt commented Jan 19, 2025

xiaoxiang781216 commented Jan 19, 2025

yamt commented Jan 20, 2025

yamt commented Jan 20, 2025

xiaoxiang781216 commented Jan 20, 2025

yamt commented Jan 20, 2025

xiaoxiang781216 commented Jan 20, 2025

yamt commented Jan 20, 2025

xiaoxiang781216 commented Jan 20, 2025

yamt commented Jan 20, 2025

yamt commented Apr 21, 2025

yamt commented Apr 21, 2025

yamt commented Apr 21, 2025

xiaoxiang781216 Apr 21, 2025

yamt Apr 21, 2025

xiaoxiang781216 Apr 21, 2025

yamt Apr 21, 2025

xiaoxiang781216 Apr 21, 2025

yamt Apr 21, 2025

xiaoxiang781216 Apr 21, 2025 •

edited

Loading

yamt Apr 22, 2025

xiaoxiang781216 Apr 22, 2025

yamt Apr 22, 2025

yamt commented Apr 21, 2025

yamt commented Apr 21, 2025

		@@ -0,0 +1,311 @@
		/****************************************************************************
		* include/errno_defs.h

generate errno definitions from CSV #15579

Are you sure you want to change the base?

generate errno definitions from CSV #15579

Conversation

yamt commented Jan 16, 2025 • edited Loading

Summary

Impact

Testing

nuttxpr commented Jan 16, 2025

yamt commented Jan 16, 2025

raiden00pl commented Jan 16, 2025 • edited Loading

yamt commented Jan 16, 2025

raiden00pl commented Jan 16, 2025

xiaoxiang781216 commented Jan 17, 2025

yamt commented Jan 18, 2025

xiaoxiang781216 commented Jan 19, 2025

xiaoxiang781216 commented Jan 19, 2025

yamt commented Jan 19, 2025

xiaoxiang781216 commented Jan 19, 2025

yamt commented Jan 20, 2025

yamt commented Jan 20, 2025

xiaoxiang781216 commented Jan 20, 2025

yamt commented Jan 20, 2025

xiaoxiang781216 commented Jan 20, 2025

yamt commented Jan 20, 2025

xiaoxiang781216 commented Jan 20, 2025

yamt commented Jan 20, 2025

yamt commented Apr 21, 2025

yamt commented Apr 21, 2025

yamt commented Apr 21, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xiaoxiang781216 Apr 21, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yamt commented Apr 21, 2025

yamt commented Apr 21, 2025

yamt commented Jan 16, 2025 •

edited

Loading

raiden00pl commented Jan 16, 2025 •

edited

Loading

xiaoxiang781216 Apr 21, 2025 •

edited

Loading