Inverted stubs are fragile in their return values

[ivg]

Dislaimer: this is not a bug, more like an interface issue, but probably it would be possible to resolve this somehow.

The inverted stubs basically follow the same principle as straight stubs and try to fit into the same interface. However, the interface similarity may hide fundamental differences, as creating a value is very different from consuming a value. Let me ground this with a concrete example. It is very easy to create the following (innocent looking) callback:

internal "lets_crash" Ctypes.(void @-> returning string) 
  (fun () -> sprintf "%g" (4. *. atan 1.))

The generated stub will drop a reference to the constructed string and after few GC cycles the data would be corrupted or a segmentation fault will happen (for the lucky ones). The same is true for other values that are managed by OCaml (i.e., block values).

It looks like, that the ctypes were originally designed with an idea of consuming values that are coming from C, and occasionally providing values to the C realm but only under a constraint that they are returned from the callback functions (e.g., the sort example). With the inverted stubs, this constraint doesn't hold anymore and returning a value that is represented as a block from the inverted stub is always a bug (unless special precautions were made to extend the lifetime of the value).

I'm not sure, whether the problem can be solved on the implementation side, or on the interface one, but at least it is good to know, that the issue exists. (And probably a big fat warning in the documentation wouldn't make any harm)

[yallop]

unless special precautions were made to extend the lifetime of the value

Right.

The recommended way to do this at present is to use the Ctypes.Root module, which supports building the sort of allocate/release interfaces that are common in C (e.g. malloc/free, fopen/fclose).

Then each OCaml value returned to C should be passed through Root.create, and later passed to Root.release when it's no longer needed, like this (untested):

  let () = internal "lets_crash_allocate" Ctypes.(void @-> returning (ptr char)) 
    (fun () -> from_voidp char (Root.create (Printf.sprintf "%g" (4. *. atan 1.))))

  let () = internal "lets_crash_free" Ctypes.(ptr char @-> returning void)
    (fun p -> Root.release (to_voidp p))

This is all quite low level (e.g. it's not very strongly typed --- see #393), but should provide the basic tools needed to build most interfaces that can reasonably be expressed in C.

I'm not sure, whether the problem can be solved on the implementation side, or on the interface one, but at least it is good to know, that the issue exists. (And probably a big fat warning in the documentation wouldn't make any harm)

Absolutely. There's reason to hope that this situation can be improved with a refined interface. In the meantime I'll mark this as a documentation issue.

[ivg]

Yeah, we're actually using the Root to return opaque pointers to OCaml values (we also added RTTI on top of root, to reduce a probability of a crash).

For the strings, that are returned from OCaml, we have a special solution: we copy them into a CArray, and store a reference to it, so that it is not GC'ed. This simplifies a lot, working with strings on the C-side.

When our solutions will get mature enough, it would be probably a good idea to port them to the upstream (ctypes).

[ivg]

Also, probably I'm already confused with two many dereferences, but it looks like that your solution should crash (hence the name 🤣):

 let () = internal "lets_crash_allocate" Ctypes.(void @-> returning (ptr char)) 
    (fun () -> from_voidp char (Root.create (Printf.sprintf "%g" (4. *. atan 1.))))

This will return a pointer that points to a pointer that points to OCaml string, not a pointer to the actual data. That's why we are using ctypes (so that we can get a real C pointer to null-terminated data)

[yallop]

Indeed; there's a level of indirection missing. I did say it was untested! Something more like this should work (still mostly untested):

 let () = internal "lets_crash_allocate" Ctypes.(void @-> returning (ptr (ptr char)))
    (fun () -> from_voidp (ptr char) (Root.create (Printf.sprintf "%g" (4. *. atan 1.))))

but it's not terribly nice. Your approach of copying into a CArray sounds like a good way of avoiding the extra indirection.

When our solutions will get mature enough, it would be probably a good idea to port them to the upstream (ctypes).

Sounds good!