erts: Skip pollset update on scheduler-polled re-arm

An enif_select fd migrated to the scheduler pollset stays registered
for IN events across triggers, so a re-arming enif_select has nothing
to change in either pollset. The scheduler pollset control was
already short-circuited for this case, but the call still fell
through to an erts_poll_control on the normal pollset: one
epoll_ctl(MOD) per wakeup that changes nothing. Skip the pollset
update entirely when re-arming a read select on an fd that is
enabled in the scheduler pollset. The trigger handler still clears
IN from state->events, so fds that stop being re-armed migrate back
to a poll thread as before.

With a pipelined request/response socket NIF workload (64 callers,
16 connections), epoll_ctl calls drop from one per wakeup (~156k
per 350k requests) to connection setup only (~240), worth ~2%
end-to-end throughput on 8 cores.

Author: lpgauth

erts/emulator/sys/common/erl_check_io.c

@@ -1409,6 +1409,17 @@ enif_select_x(ErlNifEnv* env,
 #endif
     }
 
+#if ERTS_POLL_USE_SCHEDULER_POLLING
+    if (on && ctl_op == ERTS_POLL_OP_MOD && ctl_events == ERTS_POLL_EV_IN
+        && state->flags & ERTS_EV_FLAG_IN_SCHEDULER) {
+        /* Re-arming a read select on an fd that is still registered
+         * for IN in the scheduler pollset. Neither pollset changes:
+         * IN stays enabled where it is, and the events carried by the
+         * normal pollset are not touched by this call. */
+        new_events = state->active_events;
+    }
+    else
+#endif
     if (ctl_events || ctl_op == ERTS_POLL_OP_DEL) {
 
         new_events = erts_io_control_wakeup(state,