static method hiding produces false positives when original method is forbidden

[hossman]

Consider the following code (contrived example, but simplest I could find in a hurry that anyone can try using only JDK methods w/o needing third-party libs) ...

import java.util.BitSet;
public class Main {
    public static final long[] data = new long[] {1, 2, 3, 4};
    
    public static void main(String[] args) {
        System.out.println(X.valueOf(data).toString());  // Line 6
        System.out.println(Y.valueOf(data).toString());  // Line 7
        System.out.println((new Z()).go());
    }

    public static class X extends BitSet { }

    public static class Y extends X {
        public static BitSet valueOf(long[] longs) {
            return new BitSet();
        }
    }

    public static class Z extends Y {
        public String go() {
            return valueOf(data).toString();            // Line 21
        }
    }
}

Let's see what happens when we tell forbidden-apis we don't want our code to use Bitset.valueOf(...) ...

hossman@slate:~/tmp/fapi-bug$ cat sigs.txt 
java.util.BitSet#valueOf(**)
hossman@slate:~/tmp/fapi-bug$ java -jar forbiddenapis-3.6.jar -d . -f sigs.txt 
Scanning for classes to check...
Reading API signatures: /home/hossman/tmp/fapi-bug/sigs.txt
Loading classes to check...
Scanning classes for violations...
ERROR: Forbidden method invocation: java.util.BitSet#valueOf(**)
ERROR:   in Main$Z (Main.java:21)
ERROR: Forbidden method invocation: java.util.BitSet#valueOf(**)
ERROR:   in Main (Main.java:6)
ERROR: Forbidden method invocation: java.util.BitSet#valueOf(**)
ERROR:   in Main (Main.java:7)
ERROR: Scanned 4 class file(s) for forbidden API invocations (in 0.06s), 3 error(s).
ERROR: Check for forbidden API calls failed, see log.

It correctly identified that line 6 is "bad" for it's "Forbidden method invocation", but the errors reported for Lines 6 & 21 are false positives -- those lines do not invoke Bitset.valueOf(...), they invoke Y.valueOf(...), which hides the static method with the same name provided by Y's ancestor class...

hossman@slate:~/tmp/fapi-bug$ java Main 
{0, 65, 128, 129, 194}
{}
{}

[hossman]

Practically speaking: This came up in the context of LuceneTestCase extending Assert but some Solr devs really wanting to forbid the deprecated methods in Assert -- but using static method hiding to re-define the deprecated methods in SolrTestCase still triggered forbidden api failures, so that approach was abandoned...

apache/solr#947 (comment)
https://lists.apache.org/thread/nycfk6b2bqkqsbq29snow51w31qo1dd3

[uschindler]

Hi @hossman,
sorry I was on a business trip when you opened the issue. Actually you are right the code handles static and virtual methods the same. This bug is known, but at beginning it did not seem important.

After some checking: Only the Java Compiler searches parent classes for static methods (if not found in the class referenced), but bytecode always has the exact class name. If you move a method from a class to its superclass and do not recompile the code, it fails to link the code at runtime. (incorrect, see https://docs.oracle.com/javase/specs/jls/se21/html/jls-8.html, section 8.4.8).

So basically, I will add an if statement to the method resolver to not check parent classes if the method is static.

[uschindler]

Hi, correction: Static methods are inherited by subclasses, so the behaviour is correct. The problem is the viewpoint of forbiddenapis. In contrast to some code calling a method, forbiddenapis has to make sure that also hits are found when you call a virtual method in a subclass and this one calls super.xyz(). This is the problem of real "virtual" methods. The method called is decided at runtime (therefor virtual).

For static methods, we may stop going up to superclasses, but I have to check this one more time: I am not sure if you can use super() to call superclass's static method. It makes no sense but it is unclear if the java language allows this.

If you cannot call a superclass static methods with super, I agree to remove the superclass check.

[uschindler]

I have to read https://docs.oracle.com/javase/specs/jvms/se21/html/jvms-6.html#jvms-6.5.invokestatic and compare with https://docs.oracle.com/javase/specs/jvms/se21/html/jvms-6.html#jvms-6.5.invokevirtual

May take some time.

[dweiss]

Static methods are not virtual, Uwe. They can be shadowed but the invokestatic call always uses an explicit class reference (because of that you can call SuperClass.staticMethod from Subclass.staticMethod... in fact, you can call SuperClass.staticMethod and Subclass.staticMethod from anywhere you like).

[uschindler]

Here is the way how methods are looked up: https://docs.oracle.com/javase/specs/jvms/se21/html/jvms-5.html#jvms-5.4.3

It makes no difference between static and non-static. The difference is only implemented in the compiler. Basically you can refer to a subclass' static method, if it does not exist it find the superclass' method. As forbiddenapis does not know if another subclass hides this method, it has to go to a superclass (because it needs to ensure there are no other possibilities how to call this method). Forbiddenapis problem also exists for virtual methods: As it doe snot know it the virtual method calls super.name() it has to disallow calls to that method on all subclasses. Basically there's no difference with static methods regarding lookup. If you have code that calls in its bytecode the static method Subclass#method() and this method no longer exists (e.g., because of refactopring) the JDK will per spec search for Superclass#method() [see above].

@dweiss : I explicitely said they are not virtual, I just compared the spec. When you have a virtual call you have the argument with the class on stack, but it then does exactly the same: it looks up the method as described above.

In short: actually in JVM runtime theres no difference with method lookups. The difference is only with the argument on stack which is the class for the virtual call. But lookup of method is identical, it just does not use the class on stack but the one in bytecode.

So basically that's all forbiddenapis can do. It has its limitations. We can add a specialcase for static methods to only look at the exact descriptor and not dive into superclasses, but this would have the problem of undetected calls.

[uschindler]

Ok. I think I can remove the static call. The problem with virtual calls is different. On bytecode you do not know which method is actually called, so you have to check all possible superclasses (from perspective of forbiddenapis, as first argument is unknown).

In case of static methods you can stop in forbiddenapis when method was found. The type is given on call site and you just have to look until you found one in the ancestors-or-self.

I will provide a PR.

[dweiss]

I'm not sure I agree with you here (with regard to virtual vs. static method calls)... but anyway. I think considering odd cases like swapping out classes without proper recompilation (which would emit different bytecode) is not worth it. 99.9% of the cases are legitimate valid code and javac would see the difference and link it up properly.

[uschindler]

I'm not sure I agree with you here (with regard to virtual vs. static method calls)... but anyway. I think considering odd cases like swapping out classes without proper recompilation (which would emit different bytecode) is not worth it. 99.9% of the cases are legitimate valid code and javac would see the difference and link it up properly.

See my response.

[uschindler]

Sorry for the confusion. The problem of forbiddenapis is that it needs to look somehow without runtime types. For virtual methods it is hard, because you have no information which class actually receives the call.

For static methods you can just follow the spec and stop on first found method.

[uschindler]

Btw., the same apples for fields.

[dweiss]

I did see it. But I still think maybe forbiddenapis shouldn't try to play JVM here - if a static method doesn't exist at its target class, something is wrong - classes are stale and this should perhaps produce a warning (javac should never produce such bytecode because it always resolves static call to a concrete class).