Rich mime type rendering

[SylvainCorlay]

A major next step for xeus-nelson to be really compelling would be to enable rich mime type rendering.

It would be interesting if the way to specify a representation for a custom type for xeus-nelson and xeus-octave was the same. (Presumably making use of it would require an explicit call to disp while xeus-nelson could tie deeper in the nelson AST to automatically display the last value of a cell like IPython.

cc @JohanMabille @Hind-M @rapgenic @Nelson-numerical-software.

[Nelson-numerical-software]

nelson does not support exactly same overloading than Octave. It is something that I would like to change before 1.0.

Currently, overloading is based on filename in nelson where is based on pathname for octave or matlab
(I do not consider class object, here, only overloading of types).
double_display.m vs @double/display.m

I attached a basic example to show how works overloading for double type for octave and nelson.
overload-disp.zip

extract zip, addpath and assign double or disp double variable in octave or nelson.

octave, matlab or nelson allow to overload 'disp' and 'display' functions as matlab.

For nelson part, we can use a macro or define a c++ builtin to access easier to raw values.
In nelson, internal types are not exactly organized as in octave (example complex are interleaved in nelson as m2018).

We can certainly share a common part of code to format data and others stuffs. :)

[SylvainCorlay]

The part that could be common is the way in which the rich rendering of a "pure" Octave/Nelson/Matlab object can be defined.
For example, in Python, the standard way for a class author to specify the mime bundle is to add a _repr_mimebundle_ method to the class:

def _repr_mimebundle_(self, **kwargs):
   return {
        'text/plain': repr(self),
   }

The returned value is not JSON but nested Python dicts and lists which are trivially converted to JSON.

If both xeus-nelson and xeus-octave agree which Octave/Nelson function has to be implemented and return a struct

function bundle = _jupyter_repr_(obj)
    bundle = struct(
        "text/plain", tostring(obj)
    )
end

Then xeus-nelson and xeus-octave would both consume the resulting struct in their own way to convert it to JSON for the frontend.

[rapgenic]

Hi all (developer of xeus-octave here), thank you for involving me, I definitely think this is a useful topic to discuss and standardise!

Current approach in xeus-octave

At the moment in xeus-octave for providing rich display we're overriding the display function, since it is less used than disp (and often implemented as a pure wrapper for disp). Inside the display function a display_data function can be called, with the same signature as the xeus method to actually provide the display.

This is easy, quite intuitive, very versatile and integrated. However is not standardised nor documented, so prone to breakages and arbitrary interpretations, and a bit hacky.

Pros and cons of @SylvainCorlay proposal

I actually really like the approach proposed here and thought about it in the past. The main reason I did not go for it from the beginning is that there is no way to display rich output on statements without semicolon, automatically, as octave by defaults invokes only the display function in those cases.

There are two cons as of now that I can identify:

1. The first one is the one I cited before, no rich display is done automatically unless we override the display function (I could actually try to patch octave for this, or try and mess with the parse tree of the code to achieve a similar result)
2. Converting between octave values and json is not trivial (due to missing 1 to 1 correspondence in the available types), and I'd like to be able to use octave existing implementation, which however produces a string output (whereas xeus expects a nl::JSON object). It would be nice to have xeus methods accept also string parameters instead of just json. (Note that this was already a problem regardless of this proposal, but it's a nice thing to have anyway IMO)
3. The concept might not be very familiar to octave users
4. A clear API and specification is needed (more difficult to maintain and document)

The pros:

1. Syntax similar to python, easier for those that already know the ecosystem
2. Less arbitrary, less breaking points.
3. Common convention across similar projects!!
4. Some code might actually use the display function and not like for it to be overridden, so implementing a completely new thing is IMO formally the most correct thing to do.

Other points

• Since the display function is already existing, we should also think about how to manually display those rich objects (i.e. defining an equivalent of display() python function). Something like xdisplay. An API and specification for that should be defined as well.
• We should also think about whether we want to expose low level functions like display_data and similar or if we want to handle everything with the common _jupyter_repr_ function
• We should also consider if we want a single _jupyter_repr_ function or one for each mime type (i.e. _text_plain_repr_, _text_latex_repr, etc.)

[Nelson-numerical-software]

We could serialize the types, but the deserialization has to use a convention to convert the non-finite elements and it will have a cost.

examples:

A = [1, NaN, 2; Inf, 3, -Inf];
st = jupyter_repr(A);
jsonencode(st)

ans =

    '{"class":"double","dimensions":[2,3],"issparse":false,"isreal":true,"r":[1,"Inf","NaN",3,2,"-Inf"],"im":[]}'

A = sparse(eye(3, 3) + i);
st = jupyter_repr(A);
jsonencode(st)

ans =

    '{"class":"double","dimensions":[3,3],"issparse":true,"isreal":false,"I":[1,2,3,1,2,3,1,2,3],"J":[1,1,1,2,2,2,3,3,3],"RV":[1,0,0,0,1,0,0,0,1],"IMV":[1,1,1,1,1,1,1,1,1]}'

it is not optimal but it can works
In nelson/octave, C/C++ serialization/deserialization is something regulary used with ipc, mpi, nh5, matio, mex, ...

ugly jupyter_repr.m ;)
jupyter_repr.zip