Offsets are problematic in the presence of comments

[pouryafard75]

Hello, while evaluating Spoon with cases from our benchmark, I encountered a recurring and serious issue:

//and test
class po3{
    //test
    void run(){
        // print("hello");
        print("yes");
    }

    void exe(){              
        run(52);
    }

    int a = 30;
    int b = 20;              
    
}

Leads to the following tree:

root [0,0]
    RootPac: unnamed package [0,0]
        Class: po3 [11,211]
            Method: run [26,103]
                RETURN_TYPE: void [37,41]
                Invocation: print [84,97]
                    TypeAccess: po3 [0,0]
                    Literal: "yes" [90,95]
            Method: exe [109,157]
                RETURN_TYPE: void [109,113]
                Invocation: run [143,151]
                    TypeAccess: po3 [0,0]
                    Literal: 52 [147,149]
            Field: a [163,174]
                VARIABLE_TYPE: int [163,166]
                Literal: 30 [171,173]
            Field: b [179,190]
                VARIABLE_TYPE: int [179,182]
                Literal: 20 [187,189]

The offset for Method: run [26,103] is problematic due to the presence of the comment. Specifically, the non-JavaDoc comment is unnecessarily affecting the method's offset. Since this is not a JavaDoc comment, it shouldn't influence the method's boundaries.

Notably, GumTree's JDT visitor behaves correctly, as seen here where the comment does not impact the method's offset:

CompilationUnit [0,211]
    LineComment: //and test [0,10]
    TypeDeclaration [11,211]
        TYPE_DECLARATION_KIND: class [11,16]
        SimpleName: po3 [17,20]
        LineComment: //test [26,32]
        MethodDeclaration [37,103]
            PrimitiveType: void [37,41]

I rely entirely on offsets to evaluate tools, and this minor discrepancy is causing significant conflicts. It would be very helpful to have a workaround or a fix for this issue so that comments like these do not affect method offsets.

[pouryafard75]

Also there are some subtrees, with [0,0] as offsets. Does it mean something?

[monperrus]

thanks for the bug report.

This probably comes from the advanced algorithm in Spoon to attach comments to elements. It has gone over many iterations. indeed it may be argued that the comment is semantically part of the method. If we change for your case, it may break somebody else's case.

are comments relevant in your experiment?

one option would be to diff without comment.

[pouryafard75]

@monperrus
Thanks for the answer,
Actually, I'm not concerned about validating the mappings for the comments. But we care a lot about program elements including methods, and now the offsets are different than what I expect to get.

Does Spoon have a mode that disables the parsing for the comments? This could help me get the correct offsets.
Based on my experiments, this is causing too many issues regarding the method declaration, so it is quite prevalent.

[pouryafard75]

Because I am actually running it as
new AstComparator().getCtPackage(resource)
which by default has the comments disabled but the offsets are still getting affected.

[monperrus]

which by default has the comments disabled but the offsets are still getting affected.

yes, that's probably normal.

Specifically, the non-JavaDoc comment is unnecessarily affecting the method's offset. Since this is not a JavaDoc comment, it shouldn't influence the method's boundaries.

to some extent. the reason is that Spoon tries to recover as much semantics as possible, and in many cases, the line comment just before an element is semantically part of the element itself. feel free to report a bug in Spoon directly about this.

for your experiment, one option would be to strip all comments first. WDYT?

[pouryafard75]

@monperrus
Apologies for the delayed response. We've found a workaround on our end, so this issue can be closed.
Thank you for your support!"