Understand Router and Application process messaging

[dakshinai]

Hi,

As I understand, the router process communicates to the application process via socket calls, and these calls would include the passing of parameters to invoke the application process. Can I get some direction to locate the files/functions that achieve this?

[mar0x]

Hello,

Router passes request parameters to application via shared memory. Take a look at the nxt_router_prepare_msg() function in nxt_router.c file. This function allocates shared memory buffer and fills it with various request data.

[dakshinai]

Thank you. I will begin to trace from here and follow-up if I have any more questions.

[i4ki]

Hi @dakshinai,

The high-level overview is something like:

The processes communicate by either socketpair or shared-memory, depending on the kind of message, but commonly the socketpair is used for synchronization and shm for data.
Whenever a new process is created, a new socketpair is created in the main process for the new process and when it's ready, its write file descriptor (from the pair) is sent to other processes. (See the message (NXT_PORT_MSG_NEW_PORT).

You communicate with Unit using the API maintained by the controller process (commonly listening at the unix-socket address).

The controller validates the config and send to router process (https://github.com/nginx/unit/blob/master/src/nxt_controller.c#L390).

The router parses the config accordingly, and create the resources (certificates, listeners, applications). In the case of applications, it sends a message to main process to create the user's application processes (See the message https://github.com/nginx/unit/blob/master/src/nxt_router.c#L384).

The main process forks the new process, setup the common configurations (setuid, and so on) and calls nxt_app_start where the application module is loaded (php, python, etc) and the module init function is called.
The language module commonly initialize the programming language library and finally setup the communication with the main process:
https://github.com/nginx/unit/blob/master/src/nxt_python_wsgi.c#L404

Then the module send a READY message to main, signaling that this worker is apt to receive requests. The main module will send the new process port (socketpair fd) to other interested processes (in this case just router) and the router then will be able to send synchronization messages to application process. They are received here:
https://github.com/nginx/unit/blob/master/src/nxt_unit.c#L752

Just after sending the READY message, the application module starts the main loop to receive incoming messages from router:
https://github.com/nginx/unit/blob/master/src/nxt_python_wsgi.c#L416
This is the application main loop: https://github.com/nginx/unit/blob/master/src/nxt_unit.c#L3733

Then, you can have a look in the messages NXT_PORT_MSG_MMAP and NXT_PORT_MSG_REQ_HEADERS.

As @mar0x said, have a look here to see how the request is built and sent.

As I said, this is just an overview, let me know if helps.

You can add logging calls as you did to better understand, but beware Unit heavily uses async IO,
if it doesn't get printed, maybe it doesn't get called. Also, the application API uses the nxt_unit.c
library (the function used there is nxt_unit_log or nxt_unit_debug).

[dakshinai]

Thank you. I have a couple more questions,

1. I'm not able to print values in object r, r->method e.t.c. referenced [here]

   unit/src/nxt_router.c

   Line 4877 in 36578c7

   req_size = sizeof(nxt_unit_request_t)

   )

   or print values in object field when I convert the line to nxt_log [here]

   unit/src/nxt_router.c

   Line 5005 in 36578c7

   nxt_debug(task, "add field 0x%04Xd, %d, %d, %p : %d %p",

   )

   Ideally, I'm looking at lines where the message I sent via POST is written into shared memory and the message received back as a response from the application is written back.

2. Can you elaborate on the communication between the application process and application

3. The share memory path enables the transfer of control information between the router and application process. Hence I assume, the shared memory allocation is freed. How do repeated/long-running requests work?

4. If the applications are telco workloads, can Unit still serve as a front-end to long-running telco workloads that stream data?

5. Does Unit facilitate inter-application communication with or without the router?

6. How is data isolation ensured with data from multiple apps being allocated in the shared memory area?

7. Does the app process start the application as well. During my debug sessions, I notice that I did not manually start my flask app

8. Is there a cleaner way to kill all Nginx Unit process including the app when I want to build, redeploy and test?

[dakshinai]

Any documentation on the architecture or a sync up call would be great!

[mar0x]

Hello,

1. method field of the structure nxt_http_request_t is a pointer to structure nxt_str_t. To print the value of method use %V format modifier:

    nxt_debug(task, "method = %V", r->method);

Request body copied to shared memory in this loop. src/nxt_router.c:5112 is the exact place where data is copied into shared memory.

2. I'm afraid I don't understand the question, sorry. (Web) Application is the software to process the request and produce the response. Application process is the OS entity.

3. Shared memory reused during one or multiple request processing. Application places data into shared memory, pass it to router, which sends the data to client via connection socket. Then shared memory marked as free and can be reused by same application process to pass another chunk of data.

4. For now, request body always buffered by router and passed to application as a single piece. Response data send to client once it send by application, e.g. response can be long-running.
   I have no idea how telco workloads differs from regular http request-response model, please explain.

5. Currently not. Some people asking this and we would like to know your use cases too.

6. Each application process use own shared memory segments to exchange data with router. Shared memory never shared between different application processes.

7. Yes, Unit is responsible for starting your application processes. It can also start more on demand and stop idle processes. Please read the Process Management.

8. Right now we have no such commands. You may change any application property in configuration and Unit will restart it. For instance with curl:

curl --unix-socket unit.sock -X PUT -d '"2"'  http://localhost/config/applications/my_app_name/environment/GENERATION

This will set the environment variable GENERATION=2 of application my_app_name and restart it.

[dakshinai]

Thanks for the answers. I'm from Intel. I could not find enough documentation on the internal architecture and would appreciate it if it is easier to get specific questions/use-cases addressed over a quick voice call?

Please let me know.

[i4ki]

Hi @dakshinai,

I'll let others properly answer you, but If it's only questions and use-cases, the mailing-list can be handy for longer async threads:
https://mailman.nginx.org/mailman/listinfo/unit

[dakshinai]

Thank you @mar0x and @tiago4orion

1. I did a deep dive into nxt_router_prepare_msg function. As you mentioned, I was able to see the post data in variable pos
   buf->mem.free = nxt_cpymem(buf->mem.free, pos, copy_size);

Few other constructs like the use of r->fields or p = nxt_cpymem(p, field->name, field->name_length); are not clear to me, but I'm assuming they are used for determining content length of some sort.

2. Please correct my understanding. I assumed, every application, in my case a flask web server is attached to a Unit application server? If so, how does the application server communicate with the application?

3. In cases where user requests the same data since memory is freed per transaction, we would not be able to leverage memory caching?

4. If I have a network application, that is streaming data it receives over the network, (e.g. video streaming), can NGINX unit serve such requests? If so, is the shared memory region-blocked until all write and read is complete?

5. Nothing specific, but since a shared memory region has been established, Unit could facilitate out of the box communication between two apps, I assume?

6. My question was more on, If I'm building and launching Unit from source, I will have to kill all Unit processes (main, router, controller and apps) and also clear the socket open to the controller manually

[dakshinai]

It will be great if someone can help me with a deep dive on the function nxt_router_prepare_msg, use of nxt_buf_t data structure and flow of how data write event by the router is communicated to the app process since at some point we want to carefully free the shared memory only after data is read.

[dakshinai]

Hi, @mar0x @tiago4orion any pointers when u get a chance?

[hongzhidao]

@mar0x
Just an idea of making nxt_router.c concise.
What about splitting nxt_router.c into a few small files?

nxt_router.c
It is in charge of things happened in router main/worker threads, engines, access_log.

nxt_router_conf.c
Since creating and applying conf is important in router process, it's better to separate it as an individual file.

nxt_router_req_app.c
Start processing request, including start application processes needed.

Before that, it has 5k+ lines, after splitting the file lines are.

   1156 ./src/nxt_router_conf.c
   1709 ./src/nxt_router.c
   2586 ./src/nxt_router_req_app.c

[mar0x]

Hi @hongzhidao,
I (personally) don't feel any inconvenience with large files. I never trying to count lines or looking functions with eyes and cursor keys. I'm using ctags and don't care how many files we have and how many lines each file have.
Of course, it is better to have an order and file names structure. But this question is rather to @igorsysoev because he have his own vision how it should looks like.