Type-checking of reified types

[juangamnik]

@abreslav gave me a "go for it" in:

https://youtrack.jetbrains.com/issue/KT-12897#comment=27-1519928

[abreslav]

I think we need more use cases for this. So far, it's not clear how relevant this feature might be.

[abreslav]

To be clear: this feature has rather far-fetching consequences:

• it changes the way calls are resolved (which means that such functions must be marked explicitly to avoid breaking the existing code),
• it requires some rather sophisticated front-end work, because currently inlining is done entirely on the byte code level,
• it does not make it too clear for the reader which calls are resolved in the traditional way and which are resolved in the proposed way.

So, to decide something about this proposal, we need to collect use cases and evaluate the pros and cons very carefully.

[juangamnik]

it changes the way calls are resolved (which means that such functions must be
marked explicitly to avoid breaking the existing code),

Do you think that any function relies on this behavior? The current behavior feels more like a bug and programs will just run better after doing this change.

it requires some rather sophisticated front-end work, because currently inlining
is done entirely on the byte code level,

I cannot estimate how much effort this is. I don't know exactly how Kotlin is implemented and how byte code is generated (perhaps using ASM?) Doing this inlining in the AST (or whatever intermediate data structure you use when you parse Kotlin) before it is generated to byte code is not an option or very complicated?

it does not make it too clear for the reader which calls are resolved in the traditional
way and which are resolved in the proposed way.

The behavior just changes for reified type parameters and it changes to the same behavior as it does for all other types in inlined functions. IMO being able to check with is and !is and cast values to T for returning the value implies that calling a method on an object of T will be dispatched to the type T and not to its upper bound.

The Kotlin documentation currently reads:

We qualified the type parameter with the reified modifier, now it’s accessible
inside the function, almost as if it were a normal class.

-- https://kotlinlang.org/docs/reference/inline-functions.html#reified-type-parameters

So, the more the type parameter variables typed with it behave like normal classes, the more it conforms to the expected semantics, doesn't it?

I think we need more use cases for this. So far, it's not clear how relevant this feature might be.

As I described above literally any use case for reified type parameters, like constructing the value and then executing some overriden initialization logic on it, is a use case for this proposal.

[juangamnik]

A short follow-up: I can return a value of type T from such an inline function with reified type parameter, I can even cast a value (as in the example in the kotlin documentation mentioned before) and return it. But when I call a method on it even when I cast it to T the dispatch is done statically for the upper bound of T:

(this as T) copyTo (copyAction as T)

So this is not surprising as I know that you do the dispatch during byte code generation before, but still it is a counter-intuitive and inconsequent behavior.

[ilya-g]

@juangamnik Could you clarify in the proposal, what scope is used for resolving method calls for reified types in inlined function: either it's the scope of inline function declaration, or it's the scope of the function call-site?

Could you explore, whether overload resolution conflicts become possible during inlining? How these conflicts should be reported?

[juangamnik]

@ilya-g perhaps I misunderstood you, but do you want me to describe the current situation in Kotlin or the intended situation (of the proposal)? Because if you mean the former (current situation), the answers to my issue in YouTrack suggest that it is declaration site since the byte code of the function body is copied as is. But since I have no insights (yet) to the Kotlin code, this is 2nd-hand-knowledge and it might be wrong... TL;DR I do not understand, what you exactly advice me to do 😉

[ilya-g]

@juangamnik

I misunderstood you, but do you want me to describe the current situation in Kotlin or the intended situation (of the proposal)?

As the proposal is going to change the overload resolution, the indented behavior of overload resolution should be described in the proposal.

I'll clarify the terms, so that my question is to be clear.

• Declaration-site of inline function is where the function is declared. The scope of declaration provides the symbols available inside the body of the function.
• Call-site is where the function is called. It can be in completely different module. And it has its own scope with a different set of symbols available.

So the question is which of these two scopes should be used to resolve methods calls for reified types in inline functions?
Or in terms of the example in the proposal, when you call this.copyTo(...), where (in what scope) that copyTo should be searched for?

[juangamnik]

As the proposal is going to change the overload resolution, the indented behavior of overload resolution should be described in the proposal.

I'll clarify the terms, so that my question is to be clear.

• Declaration-site of inline function is where the function is declared. The scope of declaration provides the symbols available inside the body of the function.
• Call-site is where the function is called. It can be in completely different module. And it has its own scope with a different set of symbols available.

I knew that, but better you say me something that I know, than missing something that would lead to confusion 😉.

So the question is which of these two scopes should be used to resolve methods calls for reified types in inline functions?
Or in terms of the example in the proposal, when you call this.copyTo(...), where (in what scope) that copyTo should be searched for?

Ah now I know what you mean. If the scope for available symbols could be declaration-site that would be great, because then I could call stuff in an inline method, that possible is not available at call-site (private methods) and it would behave more like a normal method definition. But I think this is out of scope of this proposal.

So this leaves call-site scope for the moment, right. And you would like me to elaborate on this and the possible conflict situations that may arise and how to handle them?

If this is what you propose (please give me a short nod or something), I can target this, but I think it will be just an explanation why this does not apply to this proposal:

• I do not propose to change the overload resolution, but the type to use for the overload resolution. It should be the reified type parameter and not the upper bound of the type parameter in the function declaration
• the resolution should stay the same

Do you agree?

I have another proposal in mind which might be fixed directly with this one regarding smart casts for reified type parameters. Currently, if you smart cast a reified type parameter(e.g. in a when expression) in an inline function, the smart casted type loses completely the information, that it has been casted from the reified type parameter, so that you have later on to cast it back to T in order to use e.g. return it as T although a cast should always be a "downcast" so that after casting to something it should definitely still be of the casted type. I think it has the same cause. The byte code in the copied inline function assumes the upper bound of T instead of the reified type parameter.

But for both cases I think the scope is irrelevant.

Do you agree?

[juangamnik]

Ah now I know what you mean. If the scope for available symbols could be declaration-site that would be great, because then I could call stuff in an inline method, that possible is not available at call-site (private methods) and it would behave more like a normal method definition. But I think this is out of scope of this proposal.

To be clear about this: I would like this very much, but I am aware, that this is nearly impossible when inlining, because after copying the code, private methods from declaration-site will just not be available (so it is a "security" issue).

On Slack language-proposals channel I proposed two more things, which I do not know whether they will be appreciated:

• allow reified type parameters on non-inline functions too. This could be done, by adding an implicit Class parameter. Of course this has not the performance impact of inline methods, but it would have some other nice properties, like being able to use recursion and use declaration-site scope... In order to have a good Java-API the compiler should add a method overload without the implicit Class parameter, which calls the method with the Class parameter with the upper bound (so there is no Uber-Ugly Java API).
• allow the alternative of copy-methods instead of inline-methods. This would mean, that instead of inlining, the whole method is copied and the type parameter is replaced by the concrete type used at the call-site. This has a performance improvement over a "normal non-inline method", since a dynamic dispatch would be replaced by a static call and it would allow recursion. This could be done even mostly automatically by the compiler as an optimization. You would not need an implicit Class parameter, but of course it adds the limitation that you have call-site scope.

I think all these 4 proposals have a common ground. Perhaps I should put all these ideas into the proposal? I didn't do that in the first place, because I was unsure, whether this would be appreciated (I wanted to start slowly 😄)

[ilya-g]

So this leaves call-site scope for the moment, right. And you would like me to elaborate on this and the possible conflict situations that may arise and how to handle them?

okay, lets stick with that option and expore it in detail. Consider the following example:

// inline function declaration-site
interface A
fun A.foo() {}

inline fun <reified T : A> invokeFooOn(instance: T) = instance.foo()


// inline function call-site
interface B
fun B.foo() {}

class C : A, B

fun test() {
    invokeFooOn(C())
}

If you call C().foo() inside test function directly it would result in overload resolution ambiguity, because A.foo() and B.foo() extensions are applicable. Now, when you call invokeFooOn and it's being inlined, it must substitute C instead of T and resolve foo method for it. Could you explain, what should happen next?

[juangamnik]

Could you explain, what should happen next?

It should happen exactly the same as if I had written this without an inline function:

// see above...

fun test() {
    C().foo()
}

The compiler will inline this and should then throw the error, that the inline function contains a call, which is ambiguous.

So this is currently OK:

fun test() {
  (C() as A).foo()
  (C() as A).foo()
}

And so this should be the behavior for the inline function:

fun test() {
  // compiler error
  invokeFooOn(C())

  // everything OK
  invokeFooOn(C() as A)

  // compiler error (because the upper bound is A and !(B: A)
  invokeFooOn(C() as B)
}

Should I add this example to the proposal?

[ilya-g]

Should I add this example to the proposal?

definitely, it would be valuable.

The compiler will inline this and should then throw the error, that the inline function contains a call, which is ambiguous.

That's where things become seeming weird: thus to use such an inline function one has to know its inner workings, its source, otherwise coping with such errors is going to be based on guesswork. And the source of the function might be not available at hand.

[juangamnik]

That's where things become seeming weird: thus to use such an inline function one has to know how its inner workings, its source, otherwise coping with such errors is going to be based on guesswork. And the source of the function might be not available at hand.

But the code won't compile without the source at hand. The compiler must have this information to do the inlining. Or is this done at runtime via byte-code weaving like in AOP?

And if he has the information he can moan. Of course, the error message has to be self-explanatory, but since the compiler has all the info...

[ilya-g]

Currently the source code isn't required to use inline functions. Inlining is performed with byte-code transformation techniques.

[juangamnik]

At runtime?

[dnpetrov]

No. Inline functions are inlined at compile-time.
Operations involving reified type parameters are annotated with special bytecode instructions.
Bytecode is post-processed at call-site, where corresponding type arguments are known.

[juangamnik]

So the information should be there to produce the compiler error and to do the "downcast", shouldn't it? For the inlining at compile time you need the source code 😄

But the downcast won't easily be done as a byte code transformation since complete method calls have to be (re)looked up and replaced. But perhaps this can be done with the annotations you mentioned that help to pinpoint these instructions and to replace them?

[juangamnik]

Sorry this is a bit off-topic, but I tried this code fiddling around with the example of @ilya-g:

// inline function declaration-site
interface A
fun A.foo() {
    print("a")
}

inline fun <reified T : A> invokeFooOn(instance: T) = instance.foo()

// inline function call-site
interface B
fun B.foo() {
    print("b")
}

class C : A, B {
    fun foo() {
        print("c")
    }
}

fun test() {
    C().foo()
    (C() as B).foo()
    invokeFooOn(C())
}

This prints out cba. The result of (C() as B).foo() baffled me. So the extension function of B wins over the member function of C? I thought an extension function would always loose against member methods. Is this a bug or intended?

[Groostav]

If dynamic makes it into the Java-side of kotlin, then you might be able to get your polymorphic extension functions with a simple

class A { }
class B : A { }

fun A.foo() = ...
fun B.foo() = ...

//...
val a = A()

(a as dynamic).foo()

--assuming of course that the dynamic cast will search for extension methods in its implementation, which, searching for things with classloaders is an oxymoron.

Does anybody know the rationale for why the C# guys chose not to implement polymorphic dispatch for extension methods? It might be valuable here. I'm looking for a fabulous adevntures in coding post but cant find one.

Looking at swift extensions, they appear statically-bound as well.

I wonder also what kinds of limitations this feature puts on type classes or other forms of ad-hoc polymorphism. The requirement that all reified extension functions must polymorphically dispatch might be difficult/impossible to maintain if/when kotlin starts getting into funny types.

Another question I have is about generics and erasure:

class A {}
class B : A {}

List<A>.foo() = ...
List<B>.foo() = ...

val bees : List<B> = listOf(B(), B())
val cast = bees as List<A> //assume kotlin.collections.List not java.util.List

cast.foo() 

erasure ensures the only thing the compiler can hit is List<A>.foo(). You might say this 'this is typical for erasure, and java programmers know how erasure works!', and you'd be right, except that by employing static dispatch kotlin allows us the two above methods, where kotlin (and java) will both forbid (or warn about ambiguity) generic implementations:

class List<T: Any>{
  fun <TA> foo(a: TA) //...
  fun <TB> foo(b: TB) //error!
}

thus, if we did add this functionality, I feel like the compiler --or rather IDEA-- would need some sophisticated "this code looks suspicious" analysis to inform the programmer what parts of variables receiver type are being bound statically (because of erasure) and what types are being bound dynamically (polymorphically).

Still, I'm with juangamnik in spirit, it seems odd that extension methods dont do anything clever with respect to overloads.

[juangamnik]

@Groostav

If dynamic makes it into the Java-side of kotlin, then you might be able to get your polymorphic extension functions with a simple
[...]
Still, I'm with juangamnik in spirit, it seems odd that extension methods dont do anything clever with respect to overloads.

I know that extension functions use static dispatch, but since C has its own implementation I thought, that the Kotlin compiler would (should) call C.foo() always and ignore the extensions. Am I wrong here? So this is not about dynamic dispatch of extension functions, but about precedence between extension methods and instance methods.

@Groostav

Does anybody know the rationale for why the C# guys chose not to implement polymorphic dispatch for extension methods? It might be valuable here. I'm looking for a fabulous adevntures in coding post but cant find one.

Xtend uses static dispatch for extension methods, but offers the dispatch keyword to allow dynamic dispatch for all parameters of a function (they implement it not via an additional V-table, but via an if/else cascade evaluating the parameters from left to right). But the extension mechanism (or at least binding extension methods) is quite different to Kotlin's so I think that a V-table-like mechanism would be necessary in Kotlin...

Further on, Xtend allows only to dispatch all parameters or none. But I think it would be great (in Kotlin too) to be able to add dispatch keyword to some parameters.

@Groostav

I wonder also what kinds of limitations this feature puts on type classes or other forms of ad-hoc polymorphism. The requirement that all reified extension functions must polymorphically dispatch might be difficult/impossible to maintain if/when kotlin starts getting into funny types.

Sorry to open the box of pandora here. I did not intend to change this proposal to how extension functions and dynamic dispatching could work. As @ilya-g said, we should focus on the dispatching of reified type parameters in inline functions.

[juangamnik]

@ilya-g I will then update the proposal according to our discussion and add this example, ok? But this may take a day or two, since I'm low on (spare) time this weekend.

[Groostav]

If dynamic makes it into the Java-side of kotlin, then you might be able to get your polymorphic extension functions with a simple

I know that extension functions use static dispatch, but since C has its own implementation I thought, that the Kotlin compiler would (should) call C.foo() always and ignore the extensions. Am I wrong here?

I'm pretty sure this is expected behaviour. The interface A doesn't define any methods, so the argument passed to your invokeFooOn has no local methods. So, following a relatively simple procedure:

• does A have an entry for foo? nope
• does A have an extension method food? yes -> use that

If A defined a local method foo, then I would expect you to get a v-table lookup and hit C's foo override.

edit: sorry I misread your comment...

but yeah, this still seems correct to me, for the same reasons that (C() as A).foo() would call the A.foo() extension function, (C() as B).foo() will call the B.foo() extension function.

The result of the expression C() as B is an instance of type B: we want to call foo on a B. There is no vtable entry for foo on the inheritence tree for B (either A or B), so it looks for extension methods. The most local extension method is B.foo().

Did you perhaps mean to rewrite interface A as

interface A {
  fun foo(){
    println("a")
  }
}

?

this will result you getting "ccc" (assuming you add an override modifier to C)

[juangamnik]

The result of the expression C() as B is an instance of type B: we want to call foo on a B. There is no vtable entry for foo on the inheritence tree for B (either A or B), so it looks for extension methods. The most local extension method is B.foo().

Somewhere in the documentation IMHO I read that extension methods should never be able to override instance methods. Since you just cast C to B with C as B and do not change the instance to be a B (it is a reference type and may be referenced by different variables). So the v-table should still be present. I think the "mistake" here is, that only the type of the variable is evaluated by the compiler and the static call is placed instead of the dynamic call to C. But since the interface B does not have a foo() method (but only an extension methods), this could perhaps be a hard task for the compiler 😐.

The only solution that comes to my mind is to do a test whether an instance method exists and then do a dynamic call (in byte code) and if not do the static call. But it is not very efficient.

So lets stick to this proposal from now on. Thanks @Groostav 😄

[abreslav]

it changes the way calls are resolved (which means that such functions must be
marked explicitly to avoid breaking the existing code),
Do you think that any function relies on this behavior? The current behavior feels more like a bug and programs will just run better after doing this change.

It is a hard constraint that the language evolves in a backwards-compatible way. This proposal will alter behavior of potentially a lot of existing code silently. We'd need very strong reasons to do that.

The claim that some behavior is "just better" than some other is an emotional statement :) Let's try to stick to technical terms. Present behavior is well-defined. It's not better or worse than any other well-defined behavior, because "better" and "worse" do not apply at this stage of language design. If we were talking about a new language feature we could maybe see what feels more natural or serves more use cases (those are arguably reasonable definitions of "better"), but at this point such reasoning is not applicable.

I cannot estimate how much effort this is. I don't know exactly how Kotlin is implemented and how byte code is generated (perhaps using ASM?) Doing this inlining in the AST (or whatever intermediate data structure you use when you parse Kotlin) before it is generated to byte code is not an option or very complicated?

It can be implemented, just very complicated. Plus, it makes more work for the type-checker, because such inline functions must be type checked on every application as opposed to once per declaration.

it does not make it too clear for the reader which calls are resolved in the traditional
way and which are resolved in the proposed way.
The behavior just changes for reified type parameters and it changes to the same behavior as it does for all other types in inlined functions. IMO being able to check with is and !is and cast values to T for returning the value implies that calling a method on an object of T will be dispatched to the type T and not to its upper bound.

Your comment does not refute my point, which is: reified or not, the code you read in a body of inline function does not type-check in place any more.

BTW, do you propose to type-check it once per declaration + per every call site, or only on call sites?

[juangamnik]

@abreslav

It is a hard constraint that the language evolves in a backwards-compatible way. This proposal will alter behavior of potentially a lot of existing code silently. We'd need very strong reasons to do that.

The claim that some behavior is "just better" than some other is an emotional statement :) Let's try to stick to technical terms. Present behavior is well-defined. It's not better or worse than any other well-defined behavior, because "better" and "worse" do not apply at this stage of language design. If we were talking about a new language feature we could maybe see what feels more natural or serves more use cases (those are arguably reasonable definitions of "better"), but at this point such reasoning is not applicable.

OK I didn't meant it in an emotional way, but that it is more consistent over the complete language (so less surprises on developer site) and more object-oriented. Of course, this is not completely objective. I saw the documentation of Kotlin saying "behaves mostly like a non-generic type" and this is why I expected, that the more it behaves like a non-generic type the more conform is it to the "specification" (I don't know whether you have some kind of language specification like Java has, where this behavior/semantics is described). So, sorry about that.

Your comment does not refute my point, which is: reified or not, the code you read in a body of inline function does not type-check in place any more.

Got your point 😄 .
My point is: inline functions behave different for reified types than for non-generic types. But if the Kotlin team thinks that this argument is not strong enough, for sure I will drop this proposal. Perhaps my last resort:

1. The underlying issue that I have with the current behavior is that if you try to use extension methods to enhance Java classes, you are doomed to static dispatching, so that any call to another extension method is not dispatched... (for reified types)
2. The most compelling example that the current behavior is not well-defined (IMHO) is that this is still not executed on current T but on upper bound for infix extension method copyTo:

(this as T) copyTo (copyAction as T)

I hope this does not sound emotional again, but a cast should lead to a type check for reified generics shouldn't it? Why bother casting it in the first place? Just for returning a value? This would be somewhat arbitrary (technically 😄).

BTW, do you propose to type-check it once per declaration + per every call site, or only on call sites?

I'm not fully confident, whether I oversee all the consequences, but "only on call sites" should suffice shouldn't it?

A last point: If one could add the keyword dispatch to extension methods, this would solve a lot of issues in this area too. I would like to see that Kotlin offers to annotate (with a modifier) complete functions (like Xtend offers), this-values of extension methods, and dedicated method parameters with this keyword, so that the corresponding parameters are dispatched like instance methods are. But this would be another proposal, which would be an additional feature not breaking any existing code. Should I add a corresponding proposal?

[juangamnik]

@abreslav If I should not drop the proposal 😉 should I rename it to "type-checking of reified types"? Since if you call an instance method on an instance of a reified type it is dynamically dispatched... So the title is currently just wrong.

[ilya-g]

I saw the documentation of Kotlin saying "behaves mostly like a non-generic type" and this is why I expected, that the more it behaves like a non-generic type the more conform is it to the "specification"

@juangamnik Do you mean this documentation: https://kotlinlang.org/docs/reference/inline-functions.html#reified-type-parameters?

The link to reified type parameter specification is given in the docs, here it is: https://github.com/JetBrains/kotlin/blob/master/spec-docs/reified-type-parameters.md

Maybe, we should fix then the documentation instead, in order not to convey false expectations? (cc: @yole)

At the moment reified type parameters behave as one could have expected them based on, say, C# experience where generics are reified.

[dnpetrov]

Ok, if we don't want to break too much, this is in fact "duck typing for reified parameters".
To do it properly, type constraints for reified types should be expressed somehow.
Otherwise, as it was said before, functions using duck typing (that is inline functions with reified type parameters) will have to be fully type-checked at call site. IMHO, that doesn't sound feasible.

NB1: it also covers leaking abstractions issue mentioned above.
NB2: there are several languages that already have that thing, F# is probably the most well-known example (although that feature is not used really often).
NB3: the original example can be rewritten differently without using features that make the code harder to understand.

Note that in Kotlin a.foo() can actually mean several really different things:
(1) function foo called with a dispatch receiver a using virtual dispatch;
(2) extension function foo called with an extension receiver a using static dispatch;
(3) member extension function foo called with an extension receiver a and some implicit dispatch receiver using virtual dispatch;
(4) function operator invoke called for a value of property a.foo, with options (1..3) for operator invoke and options (1..3) for property foo.

As of 1.1, calls with reified type parameters involved are resolved statically exactly as it happens for any other type parameters.

Now, we should specify exactly what kind of operation a.foo() is so that the function body can be type checked and bound statically, and it can be processed at call site feasibly. Compiler will resolve such calls to exactly that "reified" operation, and will substitute it with a proper call during inlining.
Pros: detailed constraint system with explicit syntax capable of specifying options (1..4) above.
Cons: simplified postponed resolve at call site.

Example (fake syntax):

typeconstraint CopyableAction<T> {
  constructor T()
  fun T.copyTo(action: T)
}

inline fun <reified T> T.copy(): T where CopyableAction<T> {
  val copy = T()
  copyTo(copy)
  return copy
}

[juangamnik]

@ilya-g

Do you mean this documentation: https://kotlinlang.org/docs/reference/inline-functions.html#reified-type-parameters?

Yes. The documentation there says:

We qualified the type parameter with the reified modifier, now it’s accessible inside the function, almost as if it were a normal class. Since the function is inlined, no reflection is needed, normal operators like !is and as are working now.

The referenced low-level description does not contain any information on how the type checking works or did I miss this?

[juangamnik]

@dnpetrov

Otherwise, as it was said before, functions using duck typing (that is inline functions with reified type parameters) will have to be fully type-checked at call site. IMHO, that doesn't sound feasible.

My proposal to use the abstract syntax tree of inline functions (instead of byte code), with placeholders (I don't have a better name) for "statements using T", that are copied to the abstract syntax tree of the call-site, replacing the type in the placeholders with the type from the call site and let the compiler do its work (it would do the full type-check then)?

Of course this does not help for leaking abstractions (you refer to @ilya-g 's example where ambiguous calls have to lead to an error on call-site. But IMHO this is exactly what I would expect. If I want reified types and I make an ambiguous call the compiler should prove me wrong 😉.

Now, we should specify exactly what kind of operation a.foo() is so that the function body can be type checked and bound statically, and it can be processed at call site feasibly. Compiler will resolve such calls to exactly that "reified" operation, and will substitute it with a proper call during inlining.
Pros: detailed constraint system with explicit syntax capable of specifying options (1..4) above.
Cons: simplified postponed resolve at call site.

Hu, this is IMHO a heavyweight solution for this. So you propose to use to "push" the extension function into some kind of explicitly defined type constraint, so that it is some kind of member extension function which leads to a virtual dispatch on the implicit receiver? How would the different implementations of copyTo be implemented? A more thorough example would then look like this in your fake syntax?:

typeconstraint CopyableAction<T> {
  constructor T()
  fun T.copyTo(action: T)
}

// should implementations be done like this?
typeconstraint CopyableTemporalAction: CopyableAction<TemporalAction> {
  fun TemporalAction.copyTo(action: TemporalAction) {
    // do the copy stuff
  }
} 

// more concrete implementations...

inline fun <reified T> T.copy(): T where CopyableAction<T> {
  val copy = T()
  copyTo(copy)
  return copy
}

[dnpetrov]

@juangamnik
That can be handled with explicit (extension) dispatching object without any reification.
Additional advantage: there's no such thing as an ambiguous implementation.

interface CopyableActionExtension<T : Action> {
  fun create(): T
  fun T.copyTo(action: T)
}

object CopyableTemporalAction : CopyableActionExtension<TemporalAction> {
  // ...
}

fun <T : Action> CopyableActionExtension<T>.copyOf(action: T): T {
  val newAction = create()
  action.copyTo(newAction)
  return newAction
}

... CopyableTemporalAction.copyOf(temporalAction) ...

[juangamnik]

@dnpetrov

That can be handled with explicit (extension) dispatching object without any reification.
Additional advantage: there's no such thing as an ambiguous implementation.

OK when I saw your code I thought already, that this can be done with an implicit receiver. So I don't get the point of your proposal with an explicit type constraint. The pro I see with your approach is, that I don't have to use the copyable explicitly. But in both cases I have to rebuild the inheritance of the original class hierarchy I want to extend. The same holds for a when expression I use currently. This is error-prone. With a full type-check (or a dispatch keyword) the original inheritance hierarchy is respected automatically.

[dnpetrov]

@juangamnik

But in both cases I have to rebuild the inheritance of the original class hierarchy I want to extend. The same holds for a when expression I use currently. This is error-prone.

Still can be done with more crafty approach to extension provider hierarchy (which can be considered boiler-plate, though).

[juangamnik]

Still can be done with more crafty approach to extension provider hierarchy (which can be considered boiler-plate, though).

It is not only boiler-plate it is redundant and error-prone or do I get the meaning of your sentence wrong?

IIRC Kotlin has been created because Java misses some language features and abstractions that make a programmers life easier as well as to overcome some bad design decisions that have been done because backward compatibility was the prime directive. IMHO solutions where I have to "copy the inheritance relation" just because I cannot convince the language to do the resolution in an object-oriented fashion were the initial driver for Jetbrains to create their own language (and because Scalas compiler was to slow). It is a common problem of many object-oriented languages that adding functionality to an existing type is a pain (leading to visitor pattern, ...) and extension methods is a nice addition to overcome this situation, but the way they are currently realized in Kotlin (as much as I like the language) I always have the feeling that I have to break my fingers.

[dnpetrov]

Indeed. That's especially true since you've mentioned Scala compilation speed. Kotlin stays away from some features for a reason.

So far, what we have:

1. Structured type constraints are preferred to non-structured.
2. It's quite unlikely that additional implicit dispatch receivers will be added to the language.

Can you think of something that will fit into these constraints :) ?

[abreslav]

@juangamnik

should I rename it to "type-checking of reified types"?

I have no objections

[juangamnik]

Adding an explicit dispatch keyword to parameters or to the extension receiver? Possible syntax:

fun dispatch SomeType.foo() { /* ... */ }

I would do it more generally to allow something like this:

// first param is dynamically dispatched, second not.
fun foo(dispatch a: A, b: B)

Further on, I would allow to define a complete method where all parameters are dynamically dispatched:

// first and second parameter are dynamically dispatched.
dispatch foo(a: A, b: B

Unfortunately, unlike in Xtend the extension receiver is written with dot-notation, so that this distinction is a bit subtle:

// extension receiver is dynamically dispatched, parameter not.
fun dispatch SomeType.foo(a: A) { /* ... */ }

// extension receiver and parameter are dynamically dispatched
dispatch fun SomeType.foo(a: A) { /* ... */ }

This would allow to explicitly decide where dynamic dispatching should be used, without the need to implement it again and again. Of course it would be nice if the dispatch keyword could be used in combination with reified types, but even if not it would already help a lot:

interface TemporalAction
interface MoveAction: TemporalAction

dispatch fun TemporalAction.copyTo(action: TemporalAction) {
  // do the copy stuff
}

dispatch fun MoveAction.copyTo(action: MoveAction) {
  // should call `TemporalAction.copyTo(TemporalAction)`
  super.copyTo(action)
  // do the copy stuff for moving
}

// more concrete implementations...

inline fun <reified T> T.copy(): T {
  val copy = T::class.java.getConstructor().newInstance()
  // this call is dispatched, because of the `dispatch` keyword in the implementations. 
  copyTo(copy)
  return copy
}

Like this is available for extension receiver there should although be a super for (explicitly) dynamically dispatched extension receivers. If there is no corresponding overload the compiler should throw an error (like for abstract parent methods).

What do you think?

[juangamnik]

@abreslav

should I rename it to "type-checking of reified types"?

I have no objections

Sorry, but does that implicitly mean I should not drop this proposal? Following @dnpetrov the proposal could change drastically because there are some constraints to Kotlin's evolution (which you mentioned before too). I tried to address these constraints in my last comments here (introducing dispatch keyword)... Does this look more promising? I am unsure how to proceed 😉. Discussing forever is not an option for anyone.

[abreslav]

@juangamnik It's up to you to decide whether to drop/abandon the proposal. My estimate so far is that it's neither easy to implement, nor a very high priority. As with any other proposal, you (and others) can work on it and refine it to a more or less doable state, and starting from there I can't predict whether it will be finally accepted (and when). It's a question of a number of factors, including:

• whether what you get in the end fits with the overall philosophy of the language,
• how many important use cases we see for the proposed feature across different real-life projects,
• how much work of higher priority we will have to do (this may be mitigated to some extent if someone contributes a good enough implementation, but most likely there will still be review/testing/integration work on our side, and it's no little time).

I tried to address these constraints in my last comments here (introducing dispatch keyword)... Does this look more promising?

If I'm not mistaken, this looks a lot like multimethods. This is a very well-studied feature which is known to cause a lot of headache implementation-wide and for one thing has very unpleasant interactions with overloading. (And it's likely a lot more work than even what you proposed initially.)

I'm very sceptical about this direction. But I'm reluctant to closing the door on any idea, who knows: maybe you can refine it so that it's great for Kotlin. Having said that, I expect it to be a lot of work on your side before this proposal becomes viable.

[juangamnik]

@abreslav

If I'm not mistaken, this looks a lot like multimethods. This is a very well-studied feature which is known to cause a lot of headache implementation-wide and for one thing has very unpleasant interactions with overloading. (And it's likely a lot more work than even what you proposed initially.)

I'm very sceptical about this direction. But I'm reluctant to closing the door on any idea, who knows: maybe you can refine it so that it's great for Kotlin. Having said that, I expect it to be a lot of work on your side before this proposal becomes viable.

Yes it is about multiple dispatch/multimethods. A lot of languages have this feature and like with many advanced features, it is not trivial to implement them and the semantics have to be chosen carefully. I will do a proposal for that as soon as I have some time and tranquillity (I hope I use this word right 😉) to do it right.

this may be mitigated to some extent if someone contributes a good enough implementation, but most likely there will still be review/testing/integration work on our side, and it's no little time

I would like to give a try to implement some of my proposals (at some point), but I need a crash course in Kotlin-development. Is there some documentation about the architecture, build-management, and how to contribute to Kotlin-code?

[abreslav]

I would like to give a try to implement some of my proposals (at some point), but I need a crash course in Kotlin-development. Is there some documentation about the architecture, build-management, and how to contribute to Kotlin-code?

@juangamnik Not very much, unfortunately. The Readme file has basic build/setup instructions. This page has some info, also there's the #kontributors channel in our public Slack

[juangamnik]

I updated this proposal regarding the discussions. Please could someone add @dnpetrov as shepherd as discussed on Slack? I will add another proposal for dynamic dispatch of extension receivers using an explicit keyword like dispatch as soon as I have some spare time.

I think this proposal is valuable as it is now, since the conflict resolution does not leak abstractions more than the already existing constraint that calling private methods in the function body of inline functions is not allowed at declaration-site (because they would not be visible at call-site where the body is copied to). Further on, IMHO this behavior change is not breaking existing source, but fixing it, since the "new" behavior respects the reified type. Compelling argument is IMO the following inconsistency:

• (a as T).foo() uses upper bound, but
• return T uses reified type, and
• b is T uses reified type.

Could you please reevaluate, whether the proposal reflects the discussions here and whether it fits into the constraints @dnpetrov mentioned?

[juangamnik]

I started a new proposal for dynamic dispatch of extension receivers as discussed with @dnpetrov here. He volunteered to shepherd this, so perhaps someone assigns the pull request to him?

I will start separate proposals for reified generics in non-inline functions (as discussed on language-proposals on slack via implicit Class-parameter or something similar) as well as one for copying generic methods to the call-site (instead of inlining) making the generic types explicit, in order to be able to "reify" the generic and still be able to use recursion, but with better performance than the implicit-parameter version as soon as I have some spare time.

[dnpetrov]

How should we handle ambiguities?

Assume you have 2 or more override-equivalent implementations of a particular function with "dispatch" receiver, possibly in different compilation units.
Here I'd expect the following:

• definitions in different compilation units not to interfere in the call resolution at run-time;
• compilation error (ambiguity) if multiple equivalent implementations are available in the same scope.

Are local "overrides" allowed for functions with "dispatch" receiver?

E.g.,

interface IA
interface IB : IA

fun (dispatch IA).bar() { ... }

inline fun <reified T> T.doBar() { 
  bar() 
}

fun IB.foo() {
  fun (dispatch IB).bar() { ... }
  doBar()
}

[juangamnik]

@dnpetrov
These seem to be questions for #46, right? Should I copy questions and answer there?

• definitions in different compilation units not to interfere in the call resolution at run-time;
• compilation error (ambiguity) if multiple equivalent implementations are available in the same scope.

I would treat both cases the same. If you import an extension from another compilation unit and get an ambiguity then, I would raise an compilation error in the importing class.

If you mean with at run-time that an existing extension method in one compilation should not be "overriden" by a definition in another compilation unit during call resolution I totally agree.

Are local "overrides" allowed for functions with "dispatch" receiver?

As a quick shot answer I would say yes. But I am unsure whether this has major implications.

[dnpetrov]

Oops, no, that's just me. Ok, let's continue under #46.

[juangamnik]

I updated some information regarding relation between #35 and #46 and a short description concerning realization.