Use post-lowering code?

[timholy]

If ExplicitImports were to iterate over all the names in the package, and then for every item that is a method, ask for Base.uncompressed_ast, then you could tally all the GlobalRefs that appear in the lowered code. This would presumably make your scope resolution fully accurate.

JuliaInterpreter does lots of shenanigans with lowered code, including replacing all GlobalRefs with the actual object (to save the cost of scope-resolution at runtime). So its code might be a good reference if you need pointers.

[timholy]

Learning materials:

• https://juliadebug.github.io/JuliaInterpreter.jl/stable/ast/
• Track 4 of https://www.youtube.com/watch?v=wXRMwJdEjX4 (roughly 1:56:30-2:14:30)

More specific code examples:

• iterating over code and finding all GlobalRefs: https://github.com/JuliaDebug/JuliaInterpreter.jl/blob/96ed09c7127475d391b1a4f20906072f482278eb/src/optimize.jl#L88-L100
• Finding all the methods defined by a package: consider using MethodAnalysis or stealing ideas from it. Particularly close to your needs is https://github.com/timholy/MethodAnalysis.jl/blob/265d33a4a2296f3f392bba9963cd8377405afabd/src/MethodAnalysis.jl#L261-L277, where you could just store Methods rather than Modules for the output, but otherwise use the same visit logic.

[ericphanson]

Thanks Tim, this is really helpful! I will give it a try.

[ericphanson]

Is there a way to get lowered top-level code in a module? I'm having trouble finding a way to do that, which would be nice since then we might be able to drop parsing altogether. (Though already using lowered code for methods would be a win).

[timholy]

You mean stuff that runs at module-definition time but doesn't result in a method signature? Great question, and AFAIK the answer is no. For that you do have to parse 😢.

You could probably call Meta.lower on it, though, if you want to use Julia's scoping to resolve the references.

[ericphanson]

Having some issues improving accuracy using lowered code:

• hard to get top-level info as discussed above. This also applies to function arguments, so using lowered information here doesn't seem to easily solve false positive in stale names with default value for function args #62
• I can't determine if a name is qualified or not from the global ref / lowered code information. So I think I need to rely on parsing, which unfortunately means in cases when parsing is difficult, I can't reliably fix it using lowered information. E.g. in False-positive stale import in Cmd interpolation #81, we miss finding unzip statically because it is not a leaf node of the syntax tree (instead the cmdstr is the leaf). In some cases (see next bullet) I can detect unzip but I can't necessarily tell if it was qualified or not. For the sake of finding a non-stale import it's OK, but I can't detect an unqualified usage of an implicitly imported name int his circumstance.
• globalrefs seem to be dynamically populated when code runs in some cases. I get different results depending on whether or not I've actually run a function, not just loaded the module is is defined in. ExplicitImports semantics are meant to be the code must be loaded but it does not have to have run. Going back to False-positive stale import in Cmd interpolation #81:

module Exporter123
export exported_a
exported_a() = "hi"
end # Exporter123

module TestInterpolation

using ..Exporter123
function register_steelProfile()
    function file()
        return print(`$(exported_a())`)
    end
end

end # TestInterpolation

and use analyze_locals_nonrecursive from my branch https://github.com/ericphanson/ExplicitImports.jl/blob/da63656eb790a8045adaac7da72367b52b27bd3c/src/analyze_lowered.jl#L38-L101, I get:

julia> lookup_lowered = analyze_locals_nonrecursive(TestInterpolation)
Dict{@NamedTuple{file::String, line::Int64, name::Symbol}, Module} with 7 entries:
  (file = "/Users/eph/ExplicitImports/test/test_interpolation.jl", line = 11, name = :cmd_gen)          => Base
  (file = "/Users/eph/ExplicitImports/test/test_interpolation.jl", line = 6, name = :_call_latest)      => Core
  (file = "/Users/eph/ExplicitImports/test/test_interpolation.jl", line = 11, name = :tuple)            => Core
  (file = "/Users/eph/ExplicitImports/test/test_interpolation.jl", line = 6, name = :TestInterpolation) => Main.TestInterpolation
  (file = "/Users/eph/ExplicitImports/test/test_interpolation.jl", line = 10, name = Symbol("#file#1")) => Main.TestInterpolation
  (file = "/Users/eph/ExplicitImports/test/test_interpolation.jl", line = 6, name = :include)           => Base
  (file = "/Users/eph/ExplicitImports/test/test_interpolation.jl", line = 6, name = :eval)              => Core

Note: no Exporter123. Then if I do

julia> TestInterpolation.register_steelProfile()()
`hi`
julia> lookup_lowered = analyze_locals_nonrecursive(TestInterpolation)
Dict{@NamedTuple{file::String, line::Int64, name::Symbol}, Module} with 9 entries:
  (file = "/Users/eph/ExplicitImports/test/test_interpo… => Main.TestInterpolation
  (file = "/Users/eph/ExplicitImports/test/test_interpo… => Base
  (file = "/Users/eph/ExplicitImports/test/test_interpo… => Base
  (file = "/Users/eph/ExplicitImports/test/test_interpo… => Core
  (file = "/Users/eph/ExplicitImports/test/test_interpo… => Main.Exporter123
  (file = "/Users/eph/ExplicitImports/test/test_interpo… => Main.TestInterpolation
  (file = "/Users/eph/ExplicitImports/test/test_interpo… => Core
  (file = "/Users/eph/ExplicitImports/test/test_interpo… => Base
  (file = "/Users/eph/ExplicitImports/test/test_interpo… => Core

I find it. Note the same thing happens without the nested function, I think it's from the command interpolation.

So I think I can improve accuracy of my scope resolution during parsing by finding globalrefs on the same line corresponding to the same name and checking their owning module, but I can't replace parsing nor add coverage to areas I don't handle well during parsing (e.g. macros, interpolation, etc).

However, I do think we can solve #73 thanks to lowered code info, at least in the case where the name in question (Transpose, being exported by LinearAlgebra and a global in the module due to @enumx) is being used in a function (so I can get the lowered info). This is a case where parsing "works", but my scope resolution is inaccurate, since it didn't recognize @enumx as declaring new globals in the module. For usages of Transpose where I can get at the lowered code, I can check the owning module (matching the parsed info against the file/line/name from the lowered code) and realize it is local to our module rather than an external name.

This isn't enough to actually close the issue, since the line @enumx ApplyStrategy Transpose Inplace will still trigger a FP where we think Transpose is coming from LinearAlgebra (since we don't have lowered code from the toplevel stuff). But we would be able to identify usages in functions.

[timholy]

I can't determine if a name is qualified or not from the global ref / lowered code information

Oof. I see why this matters, and I didn't envision that when I opened this issue.

This should become easy once JuliaLowering lands (which won't be before 1.13, and of course there are no guarantees it will happen then). I would personally be tempted to wait for that to happen before pursuing this further, but that's a personal choice.

I think it's from the command interpolation

That seems very likely. I think your visit f function should exit with return !isa(item, Method). Here's the issue: if m is a Method, then m.specializations holds all compiled MethodInstances. If you descend into them, then what you find will depend on what has been run. If you don't want that to be true, then just truncate the visitation pattern once you reach a Method.