Add new bundled signatures for disallowing internal runtime APIs + fix class loading order bugs

[rmuir]

Problem 1.
If you name a class sun.misc.Unsafe and put it in your classpath, it does not change the fact, any code calling that will still be calling the real Unsafe. But it does hide it from forbidden-apis!!!

This is because classloader order, when the application is actually used, is Bootstrap->Extensions->System (App), as explained here: http://docs.oracle.com/javase/tutorial/ext/basics/load.html

But forbidden-apis checks the wrong order, it checks the ones you provide first, because it uses lookupRelatedClass():

forbidden-apis/src/main/java/de/thetaphi/forbiddenapis/Checker.java

Lines 303 to 306 in 9de5e1f

	ClassSignature c = classesToCheck.get(internalName);
	if (c == null) try {
	// use binary name, so we need to convert:
	c = getClassFromClassLoader(type.getClassName());

This causes the interesting scenario when trying to cleanup crazy classpaths, where removing a jar can cause new violations in your build :)

Problem 2:
The current isRuntimeClass() does not seem to check for extensions at all, but only against bootstrap classpath. This hides additional internal accesses, e.g. jdk.nashorn.internal, which will cause a SecurityException if you try to use it.

So can we use the following code on pre-jigsaw, to identify extensions jars and treat them as "internal" too? I think this extensions idea goes away with jigsaw, and everything is just modules, so it should not be a problem that we can't get extensions jars/directories there.

// of course with proper checks and best-effort, not guaranteed but works.
URLClassLoader loader = (URLClassLoader) ClassLoader.getSystemClassLoader().getParent();
URL extensions[] = loader.getURLs();

Problem 3:
Internal checking has a hardcoded list of simple patterns:

forbidden-apis/src/main/java/de/thetaphi/forbiddenapis/AsmUtils.java

Line 33 in f7dce21

for (final String pkg : Arrays.asList("sun.", "oracle.", "com.sun.", "com.oracle.", "jdk.", "sunw.")) {

Can we use java.security.Security.getProperty("package.access") instead? That property is set by the JDK, in e.g. the jre/lib/security/java.security configuration file for security checks against internal apis:

#
# List of comma-separated packages that start with or equal this string
# will cause a security exception to be thrown when
# passed to checkPackageAccess unless the
# corresponding RuntimePermission ("accessClassInPackage."+package) has
# been granted.
package.access=sun.,\
               com.sun.xml.internal.,\
               com.sun.imageio.,\
               com.sun.istack.internal.,\
               com.sun.jmx.,\
               com.sun.media.sound.,\
               com.sun.naming.internal.,\
               com.sun.proxy.,\
               ... (many lines) ...

[rmuir]

By the way, i only listed these bugs because i am hitting all of them. Because we are now scanning third-party dependencies too (elastic/elasticsearch#15491), there is a lot more craziness. I am encountering crazy jar hell (with the JDK itself) and other crazy shit, and trying to get it sorted out :)

[rmuir]

I think the simplest and also faster solution is to remove all of the current 'isInternal' boot classpath logic and all of that, and just use matches(Security.getProperty("package.access") as the algorithm here. That is a totally different mechanism unrelated to module protection or anything and I would expect that its kept up to date (in my jigsaw EA builds, I have a huge list, so I think it is). It would be more portable in that sense IMO too.

[uschindler]

Hi,
I have a plan to fix this. As said before, I don't want to depend on the currently executing runtime to detect the classes (e.g. if somebody check forbidden apis with IBM J9 or a new version). The internal runtime checks should be done on the "official released JDK reference implementation" of the java version to check (not dependent on current runtime). So basically the same thing like with deprecated APIs is done here:

• add "ant generate-internal": this works like "ant generate-deprecated" and produces the signatures file. This basically parses the Security.getProperty("package.access") property and generates a simple signatures file by appendig (".**").
• deprecate the old internal-runtime-forbidden setting on forbidden apis. It wll still work like it does currently. Users of forbidden should simply use the new bundled signatures: "jdk-internal". This was already my plan, because the code to forbid packages is already there since we allow glob patterns in signatures files.

A first preview is on this branch: 727db66

[uschindler]

This patch would make #54 obsolete. I think we should simply forbid the packages and not try to detect if a class is coming from the rt.jar/module system. So we can later remove the isRuntime boolean from our class lookup.

[uschindler]

The question I have: Maybe also look at package.definition security property? The current generateor code could parse and merge both... This property has all packages that user code is not allowed to place user classes in. As we also check that our own class name is not on forbidden list, we should add both (although it looks like both flags are mostly identical in recent JVMs).

E.g., Tomcat modifies both properties to prevent bad stuff inside webapps!

[rmuir]

yeah, i know they are mutable (or configurable with simple text file). I agree to just have a baseline list for package.access.

I would not worry about package.definition, that one is totally useless! Does absolutely nothing!!!!!!!!!!!

[uschindler]

I would not worry about package.definition

My idea was to have a check on the class name of the parsed class and match it against package.definition (the same check, just statically, that Classloader does). Any class with a name from this list should not exist outside the runtime. But thats a separate thing, I agree.

[uschindler]

@rmuir about your commit: rmuir/elasticsearch@e59b990

I'd just remove the forbidden setting "internal-runtime-forbidden" (to be deprecated) and just add https://github.com/policeman-tools/forbidden-apis/blob/issues/91/src/main/resources/de/thetaphi/forbiddenapis/signatures/jdk-internal-1.8.txt as a standard signatures file to elasticsearch. So you don't need to parse log output.

This is basically what this branch is doing.

[rmuir]

Good idea! It removes some of the parsing. Unfortunately i still need parsing, for the custom whitelist handling (whitelisting missing classes and also failing the build if a whitelisted class did not have any problems)