Dealing with python object structure

[Thrameos]

Background

This is the last foundational piece that I see necessary for long term support of the JPype module. For a long time JPype has used a hack to trick the Python memory model into supporting multiple inheritance. While it looks like Python has multiple inheritance that is an illusion because it only works if the objects have no memory to back them other than a dictionary. We can try to conform to such a model which is what was the state of JPype 0.6 but it has huge pitfalls as that means hundreds of dictionary lookup per method resolution. My solution was to make a custom allocator which did functioned exactly as the alloc/free contract was designed. I add memory that I need at the end of the object outside of the view of Python and create a way to navigate to it. It is not without issues as some objects grow down meaning knowing exactly where it is for those objects takes some work, but it was very workable.

Unfortunately, Python core developers have busted their contracts in several fundamental ways. First the int contract was updated such that it didn't have a firm grasp of the size of the object. That broke us because I lost track of the end of the object. That was fixable. Next a clever Python core developer decided lets merge the Python dictionary to the end of the object. At the same time they hid the size of the dictionary and removed all of the symbols required to use the allocator. Thus they both made it hard to see how much memory I need to skip and disabled the contract that I was using without any deprecation notice nor PEP to indicate such a change was happening.

That was really the last straw for me as far as nice behavior. I have had many very important proposals such as massive speed up in the isinstance checks for derived objects or generalize support for merged exceptions that have gone nowhere because I don't have the time dedicate to Python core to write a PEP (though I was able to sign the Python contributor contract), but the core developers don't have to follow the same rules. There is a reason my avatar has been "Anger" from "Inside Out" and it isn't for my winning personality nor the fact I am a Lewis Black fan. I simply lack the demeanor to work in a politic structure.

Hence my only solution was to perform gross manipulations of the type system. There were two mechanisms in Python which are supposed to aid our usage. The first manipulated the base class size which simply runs into the same problems. The second is an Unstable API which when I tested it is broken meaning it may have worked in one version of the Python development, but the same destructive contract breaking that got me destroyed it. They have light unit testing, but it misses the broken behavior meaning they simply don't know that is nonfunctional. Python needs an overhaul of its memory model to add a layout object. Instead it has a pile of unorganized kludges with if chains checking if something gc, has inline dict, or weak reference list attachments, etc. I am not impressed. (Sorry refer to my avatar.) It also doesn't help that my work cycle only has time free in the spring quarter and never in the fall and winter which corresponds to the Python release cycle. Meaning I am forced to find solutions only when I have no time to devote to solving it.

Fundamental issue

The Python and Java tree are incompatible because Exceptions, String, Long, and Float are all related in slightly different ways. And as multiple inherence is not on the table currently, that means out of scope memory is the "easiest" solution. However, it is not the only way to tackle it.

The other way which is more challenging. If we can't add memory at object we could add it at all the leaf nodes (or at least the ones that Python allows to be extended.) Most of the object fit such a description meaning that if we are willing to accept a few slot modifications to help with the finding of the slot we can operate a bit like a relocatable dictionary.

That leaves two types in limbo. java.lang.Object and java.lang.Number. Both of these are cursed because if I add memory to them they will have base conflicts with the derived classes. And there is no way we can just not have those classes existing in Java. Though Number is not often used Object is a foundational class. Even derived classes have to go through an Object to get to the invocation stage. Meaning unless I can beat that one type into submission I am stuck with horrible kludges which will never allow us to move to alternative Python implementations.

An alternative approach

So here is my latest attempt at solving the issue. First we are going to add some memory to the _jpype._JClass type which will give us a specialized slots. As every class has a JClass instance backing it, we can store a number of hooks to help us get to our memory. I can give a slot offset which then points to the address of our memory. I will then need to implement a derived type for every Python concrete class that is in conflict. But that only solves their problem.

Next up those classes which are not part of the concrete class tree. Currently they get their memory from the JValue structure. But I can analyze the tree and run time and determine if any base class would be in conflict and if I don't find a conflict add the slot. That solves 98% of the conflicts without any need for gross hacks.

That leaves just two classes which are broken: Object and Number. My proposal is to make these "dual classes" in which we have both an abstract version which has no memory and a concrete object which holds our memory. The Python system will take any instance that was destine for the original class and vector it to our concrete class.

This isn't without pitfalls as the type of the concrete class is not the same as the type of the abstract class. Thus if someone wrote code that checked isinstance relationships using the type function on an object it would be broken. However, that seems like a very small compromise to getting rid of all of the hacks that I am currently forced to do and the ability to run on PyPy.

Request for input

@marscher @astrelsky @pelson @michi42 @altendky @Christopher-Chianelli you all contributed to the success of this package so it would help to get your opinion. This would be the first time since 0.7 that I have actually broken a contract that we have in our testbench. Though I suspect the alternative system will still be functionally equivalent, there will be code somewhere that gets broken by this change. Is is worth changing the core object memory model to make us comply with the Python object model even if it creates edge cases?

Benefits:

• Hack "free" solution. There are some classes in Python that don't have the BASE_CLASS tag that I still need to manipulate, but most will work.
• Build with PyPy
• Immunity from some level of Python core changes.

Downsides:

• More complex object model
• Different models will be present for classes meaning at least one extra function call on method resolution. Our current system is heavily optimized so one check will cost in terms of speed.
• Some objects may be forced to use the dictionary solution if I can't make them comply. Those edge cases will be slower. Clarification: this assumes that like jproxy we can keep the dict check confined, if it is required on all checks it will be cripplingly slow.
• Special is instance methods and is subclass checks to try to fool the user into not seeing extra class relationships.
• Two classes will report to the same name which can be confusing to users.
• Unknown number of packages may be broken even if our testbench passes because we can't anticipate how the model change is affects the edge cases.

[astrelsky]

I think I meantioned this once and there was a reason it wouldn't work but I don't remember. Specifically for JClass and JObject would PyObject_GetTypeData and PyObject_GetItemData work? If I remember correctly the issue was that you couldn't use them for the numeric types or exceptions.

As for Number, I think you could take advantage of the fact that they are immutable and just convert them to the standard Python equivalent with the JShort, JInt, etc. being for type hints only or just a normal python class that subclasses numbers.Number. The Number types in python also appear to be immutable, even the in place arithmetic operations return a new instance. In this case JShort, JInt, etc. can have operations that just clip the result to the java size and return a new instance. You could just implement them directly in python and discourage their use outside of typing unless absolutely necessary (ie passing an int to Java as a short).

I think you should look to comply with the Python object model and use the stable/limited API where possible. Yes there will technically be some loss in performance, nobody can argue that there is no performance difference between dereferencing a pointer vs calling a function in a shared library that does it for you. But, it shouldn't be that much of a hit and I would need to see some numbers to believe otherwise.

[Thrameos]

PyObject_GetTypeObject was supposed to address our needs as a PEP was directly in response to our appeal for an API. But it was a miss. The reason is far from obvious but has to due with that no true multi inheritance comment. When you declare that you intend to add user space it adds the basesize of the object meaning it is no different that ordinary object extension. As ordinary object extension fails due to Python refusing to allow JObject to be mixed in with int, float, char, and exception, the PEP added nothing but a redundant method of storing data in the normal structure. As the motive section started it was solving the problem I assumed that solution would address the problem, but it was a complete miss. Unfortunately they never tested if the user data could be applied to a type then type be mixed with EVERY python concrete type.

The issue with object and number (not int, float, and boxed types) is those others cannot be concrete types and have no layout. Meaning we have 2 types that can't store a java object. Everything else gets a concrete layout with an offset and these two need a completely different mechanism. If we try to store their data in a dict (JPype 0.6) then everything must consult a dict so every method resolution chain now must take the slowest possible route. Hence my solution of having two java.lang.Object classes where only the real one java.lang.(Concrete)Object stores memory.

So what breaks in the api.

jobj = somejavacall()
if isinstance(other, type(jobj)): 
   # Do something

The type returned by the type statement will be different than expected. Thus it can fail because it will be a stub rather than the abstract type. I can attempt to sew this in by overriding the instance and subinstance checks just that opens up another seam elsewhere. I can make more and more patches to cover up the lump in the carpet an pretend it isn't there, and lie to the guests that the two people with the same name tags are in fact the same person, but the mro check will clearly show one is Bob, sr. and the other is Bob, jr.

I completely agree that we should follow some Python API. Though should that be trying to get the best solution into Python by pushing users data into the preheader or creating a very complex scheme to fall into a normal model I am not sure. I tried option 1 with no traction so my only path is option 2. Unfortunately even option 2 exposes us to risk of changing as the Python developers just don't get our use case where we have to comply to the trees of two systems. They could add another speed hack to make exceptions 2% faster which works great from the Python test bench and still pulls the rug out from under our new scheme.

That is what they did when they added the embedded dictionaries to the end. It completely breaks the entire alloc and free slot mechanism and there was a completely valid method of adding it at the front like weak ref. But that is where they added it and unless they decide there is a reason to break their api a little module like our can just live with it.

Side note: the rare time Number gets used in Jpype is object conversion to mixed type number lists and BigInteger/AtomicInteger which would end up inheriting from a class with no storage. Meaning I will need a lot of test cases for every rare case that inherits from Number. (Number itself being a broken peice of the Java API where they thought abstact classes could solve everything then discovered interfaces were necessary and cowardly refused to make a scheme that an abstract class can be converted to an interface during the transition. It would have been a day for a programmer like me to make some special logic in the java 1.2 time frame and thus wrinkle would have been long forgotten but nope we just have to live with it.)

Second the downfall of this scheme happens with interfaces. You can have an exception inherit from an interface. If the interface adds memory then it will fail when the class is initialized. Meaning we don't just need a shadow for object but rather for EVERY class that can can be freely inherited. That will be A HUGE number. Of course, interfaces are special as they can be faked into the mro as they don't actually participate in most method resolutions. Just don't assume this isn't going to get messy. I tried for close to 3 months to make it work at JPype 0.7 and was forced to go down the alloc/free route. It is a huge game of wack-a-mole. It may take me many releases to find all the edge cases.

[astrelsky]

Your added memory would just be a pointer to whatever you want it to be. Couldn't you just use PyObject_GetItemData on each JClass instance which would give you the pointer to your JValue or some other abstraction?

Also I think you may as well forget about being able to subclass int or any sort of hacks. I think it is better to just convert primitives and their boxed types at the language borders. As for other Numbers in Java, such as MutableInt in apache commons, I would say it is the users problem and they can just live with having to use the methods exposed by the java class to do mutable operations on it.
https://discuss.python.org/t/using-tagged-pointers-to-support-efficient-integer-operations/87950

[Thrameos]

No the value data is per object instance as we must hold the JClass C++ type and the jvalue global reference for every java object we wrap. It does no good to have per class storage. And thus the user data API is broken. Any storage whether it be a huge struct or a single byte stops inheritance from working.

As for int and double integration that one is easy. Those types are extended in other modules enough that there are hooks and tricks that I can use to extend it. The mutable versions were never wrapped more than superficially in JPype as they are edge cases. The currently are empty objects which one can inherit from and as Number is abstract there are no instances of it that exist. But you can get something that looks like one using the cast operation. So as the subclasses that are immutable map to Python concrete types and the ones that don't become their own special cases the only worry here is I am yanking the memory from the per object and have to find a way to store it only on the abstract type. For all the concrete types the new class allocator analyzes the structure and discovers the missing data slot and bumps up the basesize in response.

Converting on the border gets us back to the string mess. That is loop back operations for containers lose type. It creates endless errors and bug reports. I won't compromise on that. But my code from 0.7 effort still works so I don't see that as an issue. All of the int types that could plague us are final meaning I can test and fix all possible holes.

The broken spot is exceptions which in Java can mix more freely than in Python. I could check if something is an exception type and delete all interfaces from it though that is my option of last resort. Interfaces on exceptions is technically doable in Java but seriously edge case as Java itself does not support catch by interface so someone must have some other reason for wanting an interface there.

[Thrameos]

With regards to tagged integers so long as the don't yank boxed type entirely I don't think it affects us. They can deal with int tagging behind the scenes all they want. We already obey the rules in that performing any operation on our object changes it to their boxed types. Converting to a tagged type is no different. Unless the plan to yank TP_BASE_CLASS from int we are safe. If they do I will join the line if people screaming bloody murder.

[astrelsky]

My mistake, you would use PyObject_GetTypeData for the value data (the reserved data for an instance of the class). I think you would need to use PyObject_GetItemData for JClass because it is type is variable sized. It is admittedly quite confusing but I do think it is useable for "normal" Java objects and classes.

https://peps.python.org/pep-0697/#relative-basicsize
https://docs.python.org/3/c-api/object.html#c.PyObject_GetTypeData

[Thrameos]

You will get the same output as this code...

class Test(Exception, int):
    pass

Test()

Which gives...

Traceback (most recent call last):
  File "/home/nelson85/conflict.py", line 1, in <module>
    class Test(Exception, int):
TypeError: multiple bases have instance lay-out conflict

Any memory added to the basesize causes conflicts. Here is the relevant code. While it doesn't mention the size directly, the first type encountered that has any memory in basicsize field is considered solid. Any other solid that is hit for which there is no direct relationship is a fail. That means PyObject_GetTypeData just won't work for our use case. I have tried several times to find any combination that is usable.

static PyTypeObject *
best_base(PyObject *bases)
{
    Py_ssize_t i, n;
    PyTypeObject *base, *winner, *candidate;

    assert(PyTuple_Check(bases));
    n = PyTuple_GET_SIZE(bases);
    assert(n > 0);
    base = NULL;
    winner = NULL;
    for (i = 0; i < n; i++) {
        PyObject *base_proto = PyTuple_GET_ITEM(bases, i);
        if (!PyType_Check(base_proto)) {
            PyErr_SetString(
                PyExc_TypeError,
                "bases must be types");
            return NULL;
        }
        PyTypeObject *base_i = (PyTypeObject *)base_proto;

        if (!_PyType_IsReady(base_i)) {
            if (PyType_Ready(base_i) < 0)
                return NULL;
        }
        if (!_PyType_HasFeature(base_i, Py_TPFLAGS_BASETYPE)) {
            PyErr_Format(PyExc_TypeError,
                         "type '%.100s' is not an acceptable base type",
                         base_i->tp_name);
            return NULL;
        }
        candidate = solid_base(base_i);
        if (winner == NULL) {
            winner = candidate;
            base = base_i;
        }
        else if (PyType_IsSubtype(winner, candidate))
            ;
        else if (PyType_IsSubtype(candidate, winner)) {
            winner = candidate;
            base = base_i;
        }
        else {
            PyErr_SetString(
                PyExc_TypeError,
                "multiple bases have "
                "instance lay-out conflict");
            return NULL;
        }
    }
    assert (base != NULL);

    return base;
}

Also PyUnstable_Object_GC_NewWithExtraData which should be our go to is also broken. While it reports extra memory was added for our use that memory gets overwritten meaning somehow the GC has lost the size of the object.

PyObject *
PyUnstable_Object_GC_NewWithExtraData(PyTypeObject *tp, size_t extra_size)
{
    size_t presize = _PyType_PreHeaderSize(tp);
    PyObject *op = gc_alloc(_PyObject_SIZE(tp) + extra_size, presize);
    if (op == NULL) {
        return NULL;
    }
    memset(op, 0, _PyObject_SIZE(tp) + extra_size);
    _PyObject_Init(op, tp);
    return op;
}

[Thrameos]

So here is the Plan.

We need 2 pointers to make this work. One the class structure and one that us a union type jvalue. Currently both appear in our hidden memory. We derive from object, type, long, float, unicode and exception. The current solution allows arbitrary mixins from any type.

We can move one pointer from the object to the type. That just leaves the jvalue. For thus we only access the jvalue once per call (vs type piece that gets many hits every method resolution.). Therefore we can also add a private slot to the type object which describes how to find the jvalue in the object. For ordinary classes we csn just use offsetof. For int we would need to add to the item area with a custom allocator. This just leaves mixing properties (an exception that implements an interface) For those we added to dual object where we analyze if s type is missing space and derive it to get the proper concrete type.

This is a lot of pain to implement but plausible.

[Christopher-Chianelli]

If I understand correctly:

• JPype uses a custom allocator that allocates extra space at the end for internals
• CPython devs decide to change the structure of objects making it hard/impossible to determine where their data ends
• CPython devs also decide to remove symbols that are needed in order to use a custom allocator
• New proposal is to add CPython slots to JClass with each subclasses adding their own custom slots for their fields/methods/extra data

The new solution sounds good if it works. I remember working on a CPython patch that tries to discover used fields on a new class and automatically create a __slots__ from them (with an extra __dict__ slot at the end); that being said, I recall running into some issues that ultimately made it impossible in the general case.

If a custom allocator was still possible and the only issue was discovering where an object ends, I would move JPype data BEFORE the CPython object structure. For example, if a CPython object needed 32 bytes, JPype internals need 8 bytes:

• The custom allocator would allocate 40 bytes at a pointer p
• JPype will store it data in the first 8 bytes of p
• The custom allocator would return q = p + 8
• In order to read/write data from a CPython pointer, it would read/write from q - 8

The only potential issue is a realloc which resizes the pointer; I am unsure if CPython allocate API allows redefining realloc so it knows the real pointer is at q - 8.

[Thrameos]

Thanks for the input! The problem goes beyond not being able to determine the end but rather they have added an undefined sized object that starts immediately after the end of the current object model. As it is both a hidden definition and variable in length it is very hard to deal with. In addition they have removed the alloc gc methods meaning that you can’t just make an extra blob of memory and tell them to consider it part of the GC system. That is why I think they really messed up their alloc/free slot system with those changes. Unless you are doing something with a pool and want fully custom stuff the API is no longer useful. Moving ahead would seem like the next logical choice (in fact it was my first choice), but that is impossible. They add gc, weak, and dict offset ahead of the user exposed model (and logically that would also seem where that private inline dict should have been) and they did not give a slot to how much is ahead. Instead they compute it on the fly from the flags. That means when you free it rather than just `free(object-offset)` they do `free(object + calc(flags))` so if we add memory we free the wrong address. (Again this is why the piecemeal approach they are taking seems very bad to me. Rather than just biting the bullet and deciding a safe way to support now and old methods with a little extra memory, they are incrementally adding features and increasing the burden on their system. Unless they step back and fix the bad design choices they are going to need Python 4.0) My current idea requires adding memory where we can (a normal object), using the item section where we can’t and making it so we refuse to add a dictionary. But I am still trying to prove it handles all case.