confusing treatment of context function literal

[symingz]

Compiler version

3.4.2

Minimized code

File1:

def foo() =
  (x: Int) ?=> x + 1

File2:

def foo() =
  println()
  (x: Int) ?=> x + 1

Output

File1: no error

File2:

-- [E172] Type Error: Main.scala:3:20 ------------------------------------------
3 |  (x: Int) ?=> x + 1
  |                    ^
  |    No given instance of type Int was found for parameter of (Int) ?=> Int
1 error found

Expectation

Both files should compile. Or both files should fail to compile.

[som-snytt]

Similar to #19021

https://docs.scala-lang.org/scala3/reference/contextual/context-functions.html

Context functions require that the expected type is a context function and the expression is a context function literal. Otherwise a context function value gets applied.

Maybe the doc needs to be more explicit, and also show what does not work.

Maybe the "implicit function types" blog could be updated and added to the docs.

There are two rules that guide type checking of implicit function types.

references to implicit functions get applied to implicit arguments in the same way as references to implicit methods.

The second rule is the dual of the first. If the expected type of an expression t is an implicit function type then t is converted to an implicit closure, unless it is already one.

On these two commandments hang all the law and the prophets.

The broader context is that the feature is a way to simplify the syntax for passing implicit context. It is not merely a different way of writing { implicit x => } anonfun literals. (That is a feature people ask for back.) In this sense, it does not cater to folks writing explicit arrows, which is to restore boilerplate.

Leaving this ticket open for the doc enhancement. Also tagging the Scala 3 spec for adding this information, and also removing the paragraph on implicit x as noted by the current reference.

[symingz]

"Context functions require that the expected type is a context function and the expression is a context function literal." makes sense to me. But even with that in mind, I still find the compiler's behavior confusing.

In particular, neither File1 nor File2 specifies foo's return type, so "the expected type is a context function" is false in both cases. Yet File1 compiles and File2 doesn't.

Furthermore, if I change File2 to

def foo(): (Int) ?=> Int =
  println()
  (x: Int) ?=> x + 1

to make it clear that a context function is expected, compilation fails with this error:

-- [E007] Type Mismatch Error: Main.scala:3:2 ----------------------------------
3 |  (x: Int) ?=> x + 1
  |  ^^^^^^^^^^^^^^^^^^
  |  Found:    (Int) ?=> Int
  |  Required: Int
  |
  | longer explanation available when compiling with `-explain`
1 error found

Which doesn't make sense to me. Where does Required: Int come from?
In contrast, a similar change to File1 is totally fine.

Additionally,

def foo() =
  val f = (x: Int) ?=> x + 1
  f(using 10)

compiles fine. But in val f = (x: Int) ?=> x + 1, there is no expectation for f to be a context function either (no explicit type is given to f), so compilation should fail with a similar message as File2 (the first version), to be consistent.

At the risk of side tracking, I would like to mention another confusing behavior, because it somehow seems related:

def foo() =
  given Int = 10
  val f = (x: Int) ?=> x + 1
  f

compiles. But

def foo() =
  given Int = 10
  (x: Int) ?=> x + 1

fails with

-- [E172] Type Error: Main.scala:3:20 ------------------------------------------
3 |  (x: Int) ?=> x + 1
  |                    ^
  |    No given instance of type Int was found for parameter of (Int) ?=> Int
1 error found

[som-snytt]

Example for inferring a context function when the RHS is a context function literal: https://github.com/scala/scala3/blob/main/tests/run/implicitFuns.scala#L8

It's not sufficient that the RHS is a block that may later turn out to have a context function result expression.

I, too, made up a brief catalog.

I wrote: "There must be a pop quiz on Coursera that asks which of these compiles as expected?"

I retracted the comment, but I think it's funny because one wouldn't expect this to be that tricky. (As I'm sure you agree.) Maybe I'm wrong that this is only a doc issue; maybe they've come up with an improvement in ergonomics.

def foo(): Int ?=> Int =
  println()
  (x: Int) ?=> x + 1

def bar() =
  (x: Int) ?=> x + 1

def baz() =
  println()
  (x: Int) ?=> x + 1

def buz(): Int ?=> Int =
  println()
  val f: Int ?=> Int = (x: Int) ?=> x + 1
  f

def f: Int ?=> Int = (x: Int) ?=>
  println()
  x + 1

buz is your side-tracking example, but it doesn't compile as I would expect, since it does not return f unmolested. It has an extra closure, but I see that is eliminated in erasure, so, "no harm no foul".

The useful option is -Vprint:parser,typer. The docs emphasize that the rewrites happen early, that is, before type checking; for example, invoke(e) becomes invoke(x ?=> e) to make the implicit arg available in e.

foo (which is when you naturally give it an explicit CF return type) is actually

    def foo(): (Int) ?=> Int =
      {
        def $anonfun(using contextual$1: Int): Int =
          {
            println()
            {
              def $anonfun(using x: Int): Int =
                {
                  x.+(1)
                }
              closure($anonfun)
            }
          }
        closure($anonfun)
      }

where the "outer" function wrapper must return an int
and the difference with buz that typechecks:

    def buz(): (Int) ?=> Int =
      {
        def $anonfun(using contextual$2: Int): Int =
          {
            println()
            val f: (Int) ?=> Int =
              {
                def $anonfun(using x: Int): Int = x.+(1)
                closure($anonfun)
              }
            f.apply(contextual$2)
          }
        closure($anonfun)
      }

[som-snytt]

def foo() =
  given Int = 10
  (x: Int) ?=> x + 1

has the apply inserted at the block (!).

    def foo: <error no implicit values were found that match type Int> =
      {
        final lazy given val given_Int: Int = 10
        {
          def $anonfun(using x: Int): Int =
            {
              x.+(1)
            }
          closure($anonfun)
        }
      }.apply(/* missing */summon[Int])

that is the same as baz, but is a more interesting puzzler during happy hour.

[symingz]

Thank you for the explanation. But I still can't deduce a clear set of rules which I can use to reason about the compiler's behavior. Most importantly, why is apply(summon[Int]) or apply(contextual$2) added in some cases and not others?

[som-snytt]

That is rule one. (There are two rules.)

def f(i: Int)(implicit ord: Ordering[Int]) = ???
f(42)
f(42)(Ordering_Int)

analogously

def f: Int ?=> Int = ???
f
f(given_Int)

compare with

scala> def f(i: Int)(implicit ord: Ordering[Int]) = i+1
def f(i: Int)(implicit ord: Ordering[Int]): Int

scala> Option(summon[Ordering[Int]]).map(f(42))
val res0: Option[Int] = Some(43)

where f(42) is applied or not depending on the expected type.

The details page in the reference says:

Context function literals (x1: T1, ..., xn: Tn) ?=> e are automatically created for any expression e whose expected type is scala.ContextFunctionN[T1, ..., Tn, R], unless e is itself a context function literal. This is analogous to the automatic insertion of scala.Function0 around expressions in by-name argument position.

The analogy holds in the other direction, where p reference to a by-name parameter usually means evaluation, except in eta-expansion.

So rule two is when apply is not inserted. It says your ordinary expression is wrapped in a context function where that is expected ... unless it is already a context function literal, which is the case you ask about.

Probably one of the usual suspects will come along to provide clarifying insight.

[symingz]

OK, I think I understand now.

The Rules

1. A context function literal always simply produces a context function value, no matter the context of use.
2. In a context where a context function is required/expected (based on explicitly/user provided type signatures), an expression which is not a context function literal will be rewritten to be wrapped inside an auto-generated context function literal. This is done even if the expression's type is a context function (but the expression is not a context function literal).
3. In other contexts, if an expression has a context function type, but is not a context function literal, then it is rewritten such that an invocation of "apply(summon[...])" is attached to it.

Note that when a block expression appears in a Rule2-context, the context DOES NOT propagate to the block expression's last sub-expression. For example, in

def foo(): (Int) ?=> Int =
  given Int = 10
  val f = (x: Int) ?=> x + 1
  f

Because of the explicit return type for foo which asks for a context function, the block expression {given Int = 10; val f = (x: Int) ?=> x + 1; f} will receive a Rule2-rewrite. But f, the last sub-expression, which is a context function but not a literal, will receive a Rule3-rewrite.

Case 0

def foo() =
  (x: Int) ?=> x + 1

def foo(): (Int) ?=> Int =
  (x: Int) ?=> x + 1

Both compile because of Rule 1.

Case 1

def foo() =
  println()
  (x: Int) ?=> x + 1

In this case, the compiler encounters TWO expressions with context function type. First the compiler comes across (x: Int) ?=> x + 1. The compiler leaves it be (Rule 1). As such, the compiler then comes across { println(); (x: Int) ?=> x + 1 }, which has type (Int) ?=> Int, so the complier rewrites it to { println(); (x: Int) ?=> x + 1 }.apply(summon[Int]) (Rule 3). Even knowing the rules, this could be surprising, because one may not realize that the multi-line body of a method is in fact an expression. But it is, and thus we get the compilation error:

-- [E172] Type Error: Main.scala:3:20 ------------------------------------------
3 |  (x: Int) ?=> x + 1
  |                    ^
  |    No given instance of type Int was found for parameter of (Int) ?=> Int
1 error found

The error message unfortunately adds to the confusion. As it makes it seem like the compiler has added apply(summon[Int]) to the literal. But really the compiler has added apply(summon[Int]) to the block expression.

Case 2

def foo(): (Int) ?=> Int =
  println()
  (x: Int) ?=> x + 1

Similar to Case 1, except { println(); (x: Int) ?=> x + 1 } is now in a context where a context function of type (Int) ?=> Int is expected, thus Rule 2, not Rule 3, is triggered. So the block expression is rewritten to (_: Int) ?=> ({ println(); (x: Int) ?=> x + 1 }: Int). Hence the compilation error:

-- [E007] Type Mismatch Error: Main.scala:3:2 ----------------------------------
3 |  (x: Int) ?=> x + 1
  |  ^^^^^^^^^^^^^^^^^^
  |  Found:    (Int) ?=> Int
  |  Required: Int
  |
  | longer explanation available when compiling with `-explain`
1 error found

Once again, in addition to the rules, it is key to realize that method foo's whole body is a block expression. As such, the expectation of an (Int) ?=> Int context function as foo's return value is applied to the block expression (but not to the last expression (i.e. the literal) within the block, as explained in The Rules)

Case 3

def foo() =
  given Int = 10
  val f = (x: Int) ?=> x + 1
  f

Two expressions with context function types. First (x: Int) ?=> x + 1, no rewrite (Rule 1). Then f, Rule 3 adds apply(summon[Int]). As such, the block expression { given Int = 10; val f = (x: Int) ?=> x + 1; f } has type Int, and is in an unconstrained context, so no more rewrites. Compilation passes.

Case 4

def foo() =
  given Int = 10
  (x: Int) ?=> x + 1

Two expressions with context function types. First (x: Int) ?=> x + 1, no rewrite (Rule 1). Then { given Int = 10; (x: Int) ?=> x + 1 }, Rule 3 adds apply(summon[Int]). Since apply(summon[Int]) is outside the block, it cannot see given Int = 10, thus the compilation error:

-- [E172] Type Error: Main.scala:3:20 ------------------------------------------
3 |  (x: Int) ?=> x + 1
  |                    ^
  |    No given instance of type Int was found for parameter of (Int) ?=> Int
1 error found

Rules Changes to Reduce Confusion

If the rules are changed to the following, the compiler's behavior may be less confusing.

1. A context function literal always simply produces a context function value, no matter the context of use.
2. In a context where a context function is required/expected (based on explicitly/user provided type signatures), an expression which is not a context function literal will be rewritten to be wrapped inside an auto-generated context function literal. This is done even if the expression's type is a context function (but the expression is not a context function literal).
3. In other contexts, if an expression has a context function type, but is not a context function literal or block expression, then it is rewritten such that an invocation of "apply(summon[...])" is attached to it.