Interacting with non-langium models

[goto40]

I am playing with langium 1.1.0 and I wish to create a model which is interacting with some other model, assuming I have a typescript representation of the other model.

As an example I use JSON as the "foreign" meta-model and JSON-files.

My langium garmmar is then sketched as follows:

grammar HelloJson

entry Model:
    (includes+=JsonInclude
    | jsonReferences+=JsonReference
    )*;

JsonInclude:
    'include-json' filename=STRING 'as' name=ID;

interface JsonElement {
    fqn: string             // name
    stringValue: string // (unused)
}

JsonReference:
    'reference' include=[JsonInclude]
    ':' element=[JsonElement:FQN]
;

FQN returns string: ID ('.'FQN)*;
// ... rest is the same as the hello-world-grammar

I use the artifical AST Nodel "JsonElement" to represent the link to JSON files.

I add a scope provider to implement the lookup and linking:

export class HelloJsonScopeProvider extends DefaultScopeProvider {

    jsonFileData = new Map<string, Array<JsonElement>>();

    override getScope(context: ReferenceInfo): Scope {
        const referenceType = this.reflection.getReferenceType(context);
        if (isJsonReference(context.container) && referenceType===JsonElement && context.property==='element') {
            const jsonReference = context.container;
            const elements = this.loadJsonFile(jsonReference);
            const descriptions = stream(elements).map(element =>
                this.createJsonDescription(element, jsonReference));
            return new StreamScope(descriptions);
        }
        else {
            return super.getScope(context);
        }
    }

    createJsonDescription(element: JsonElement, jsonReference: JsonReference): AstNodeDescription {
        return {
            node: element,
            type: JsonElement,
            name: element.fqn,
            documentUri: getDocument(jsonReference).uri, // TODO?
            path: '' // TODO?         
        }
    }

    loadJsonFile(jsonReference: JsonReference): Array<JsonElement> {
        const elements = new Array<JsonElement>;

        const fileName = jsonReference.include.ref?.filename;
        if (fileName===undefined) throw new Error(`unexpected: unresolved reference to ${jsonReference.include.$refText}`);
        if (this.jsonFileData.has(fileName)) {
            const result = this.jsonFileData.get(fileName);
            if (result===undefined) throw new Error('unexpected');
            return result;
        }

        const data = JSON.parse(fs.readFileSync(fileName, 'utf-8'));
        if (typeof data === 'object') {
            rec('', data);
        }

        function rec(baseName: string, data: object): void {
            for(const [key, value] of Object.entries(data)) {
                const fqn = baseName+key;
    
                if (typeof value === 'object' && value!==null) {
                    rec(fqn+'.', value);
                }
                else {
                    const doc = jsonReference.$document; // TODO?
                    elements.push({
                        $type: JsonElement,
                        fqn: fqn,
                        stringValue: 'todo-stringValue',
                        $document: doc
                    });        
                }
            }
        }

        return elements;
    }
}

With this I manage to get a correct completion-and-linking behavior:

my model:

include-json "./hello.json" as hello

reference hello: inner1.inner11.name
reference hello: name

my external hello.json:

{
    "name": "Pierre",
    "type": "Person",
    "id": 123,
    "inner1": {
        "innername": "cool1",
        "innerinfo": "info",
        "inner11": {
            "name": "Pierre2",
            "type": "Person2"
        }
    }
}

However,

• I get errors in the my output window: ' Request textDocument/documentHighlight failed.
  Message: Request textDocument/documentHighlight failed with message: AST node has no document.
  Code: -32603 '
• I also have no idea where to continue to enable "F12" (goto definition), assuming I know the location of JSON elements in the JSON file, and the JSON file URL. Where would I handle this?
• Is this the correct way? Or should I follow a complete other way to implement such linking to external models?

[msujew]

Is this the correct way? Or should I follow a complete other way to implement such linking to external models?

There are ways to accomplish what you're trying by creating synthetic nodes, but it requires some effort. Especially if you really want to deal with all the LSP stuff correctly. It would require overriding every service.

I would instead recommend to model the JSON spec in Langium (it should be fairly trivial) and register that as a second language in your language server. Not only removes that the need for manual transformation of JSON to langium, but also makes LSP integration a breeze. Note that you shouldn't register the additional language in vscode, since it already has builtin JSON language support.

[goto40]

I understand. Thank you for the quick resonse. So probably this is too complicated.

However, I wonder if not more people have the need to combine their model with external sources (JSON is just a simple example of such a model).

I if model the link to my external model as payload (not as langium reference):

JsonReference:
    'reference' include=[JsonInclude]
    ':' element=FQN
;

instead of

JsonReference:
    'reference' include=[JsonInclude]
    ':' element=[JsonElement:FQN]
;

I could perfectly validate my model concerning correctness.

--> How difficult would it be to implement completion and goto-reference (less services to reimplement? is it possible to name these services?)?

--> is this a better solution compared to the previously formulated?

[goto40]

I played with the CompletionProvider in order to see if the idea w/o langium-referenes could work. At many places this was trial-and-error...

I modified my grammar (to represent the links/references JsonReference.jsonLink to my external model as strings):

grammar HelloJson

entry Model:
    (includes+=JsonInclude
    | jsonReferences+=JsonReference
    )*;

JsonInclude:
    'include-json' filename=STRING 'as' name=ID;

JsonReference:
    'reference' include=[JsonInclude]
    ':' jsonLink=FQN   // <----- This is my external reference
;
//...

I defined and registered a CompletionProvider (here, the way how I decide when to create my customized completion hints was very experimental / I figured out with printfs what condition to use)

export class HelloJsonCompletionProvider extends DefaultCompletionProvider {

    externalModel: ExternalModel = new ExternalJsonModel();

    protected override completionFor(context: CompletionContext, next: NextFeature, acceptor: CompletionAcceptor): MaybePromise<void> {
        if (isJsonReference(context.node) && next.feature.$containerProperty==='elements') {
            const include = context.node.include.ref;
            if (include!==undefined) {
                const url = new URL(include.filename, include.$container.$document?.uri.toString()).toString();
                const completionValues = this.externalModel.getData(url);
                completionValues.forEach(v=>{
                    acceptor({detail: v.fqn, kind: 1, label: v.fqn});
                })
            }
        }
        return super.completionFor(context, next, acceptor);
    }
}

and I added a validator to validate my external "references" (represented as string).

export class HelloJsonValidator {

    constructor(private services: HelloJsonServices) {}

    checkJsonReference(r: JsonReference, accept: ValidationAcceptor): void {
        const externalModel = (this.services.lsp.CompletionProvider as HelloJsonCompletionProvider).externalModel;
        const include = r.include?.ref;

        if (include){
            const url = new URL(include.filename, include.$container.$document?.uri.toString()).toString();
            if (!externalModel.isReferenceValid(url, r.jsonLink)) {
                accept('error', `unresolved ${r.jsonLink}`, {node: r, property: 'jsonLink'});
            }
        } 
    }

}

With this I got some basic (very limited) preliminary working results:

I implemented the external model with a JSON file (as a simple example of an external model - it could be any model we have a library for; With node-jsonc-parser I get location information required for the goto-defintion functionality sketched below):

import * as fs from 'fs'
import { LocationLink } from 'vscode-languageclient';
import { JSONPath, visit } from 'jsonc-parser';

export interface ExternalModelData {
    fqn: string
}
export interface ExternalModel<Data extends ExternalModelData=ExternalModelData> {
    isReferenceValid(url: string, fqn: string): boolean;
    getData(url: string): Array<Data>
    getDefinitionLocation(url: string, fqn: string): LocationLink;
}

// more details: see blow

With an additional DefinitionProvider, I also get support for F12 (goto definition):

import { CstNode, DefaultDefinitionProvider } from "langium";
import { DefinitionParams, LocationLink } from "vscode-languageclient";
import { isJsonReference } from "./generated/ast";
import { HelloJsonCompletionProvider } from "./hello-json-completion";
import { HelloJsonServices } from "./hello-json-module";

export class HelloJsonDefinitionProvider extends DefaultDefinitionProvider {

    constructor(private services: HelloJsonServices) {
        super(services);
    }

    protected override async collectLocationLinks(sourceCstNode: CstNode, params: DefinitionParams): Promise<LocationLink[] | undefined> {
        const externalModel = (this.services.lsp.CompletionProvider as HelloJsonCompletionProvider).externalModel;
        const result = await super.collectLocationLinks(sourceCstNode, params);
        if (result===undefined && isJsonReference(sourceCstNode.element)) {
            const include = sourceCstNode.element.include.ref;
            //console.log(`preparing goto "${sourceCstNode.element.jsonLink}"`);            
            if (include!==undefined) {
                const url = new URL(include.filename, include.$container.$document?.uri.toString()).toString();
                //console.log(`-> goto "${sourceCstNode.element.jsonLink}/${url}"`);
                return [externalModel.getDefinitionLocation(
                    url,
                    sourceCstNode.element.jsonLink
                )];
            }
        }
        return result;
    }
}

My refined question, thus, is:

• Is it a "good" way to implement external links in such a manner? Is it likely to break in future langium releases?
• Is the implementation of HelloJsonDefinitionProvider somewhat "ok"? It seems to me, that the solution is limited to simple rule layouts: in collectLocationLinks (see above), if super.collectLocationLinks return undefinedand we have a JsonReference then build the list of locations.
• Is the implementation of HelloJsonCompletionProvider somewhat "ok"? It seems to me, that the solution is limited to simple rule layouts: in completionFor (see above), if we have a JsonReferenceand next.feature.$containerProperty==='elements' (result of trial-and-error), then build the list of completion values.

@cdietrich I would appreciate to have some information from your experiments - I think you might probably have a much more elaborated solution to implement links to external models.

More details for the external model implementation:

// more details (continued; see above):

export interface ExternalJsonModelData extends ExternalModelData {
    location: LocationLink;
}

export class ExternalJsonModel implements ExternalModel<ExternalJsonModelData> {
    jsonFileData = new Map<string, Map<string, ExternalJsonModelData>>();

    isReferenceValid(url: string, fqn: string): boolean {
        const data = this.getData(url);
        return data.some(d=>d.fqn===fqn);
    }

    getDefinitionLocation(url: string, fqn: string): LocationLink {
        const model = this.jsonFileData.get(url);
        if (model===undefined) throw new Error(`${url} not an external model`);
        const result = model.get(fqn);
        if (result===undefined) throw new Error(`${fqn} not in external model ${url}`);
        return result.location;
    }

    getData(url: string): ExternalJsonModelData[] {
        const fileName = new URL(url).pathname;
        const elements = new Array<ExternalJsonModelData>;

        if (this.jsonFileData.has(fileName)) {
            const result = this.jsonFileData.get(url);
            if (result===undefined) throw new Error('unexpected');
            return Array.from(result.values());
        }

        let current: string|undefined|number;
        const names = new Array<string|number>;
        function onObjectProperty(property: string, offset: number, length: number, startLine: number, startCharacter: number, pathSupplier: () => JSONPath): void {
            current=property;
        };
        function onObjectBegin(offset: number, length: number, startLine: number, startCharacter: number, pathSupplier: () => JSONPath): void {
            if (current!==undefined) {                
                names.push(current);
                current=undefined;    
            }
        };
        function onObjectEnd(offset: number, length: number, startLine: number, startCharacter: number): void {
            current = names.pop();
            if (typeof current === 'number') current++;
        };
        function onArrayBegin(offset: number, length: number, startLine: number, startCharacter: number, pathSupplier: () => JSONPath): void {
            if (current!==undefined) {
                names.push(current);
                current=0;
            }
        }
        function onArrayEnd(offset: number, length: number, startLine: number, startCharacter: number): void {
            current = names.pop();
        }

        function onLiteralValue(value: any, offset: number, length: number, startLine: number, startCharacter: number, pathSupplier: () => JSONPath): void {
            function f(x:string|number) {
                if (typeof x === 'number') return `_${x}`;
                else return x;
            }
            const fqn = `${names.map(f).join('.')}${names.length>0?'.':''}${current}`;
            elements.push({
                fqn: fqn,
                location: {
                    targetUri: url,
                    targetRange: {start:{line:startLine,character:startCharacter},end:{line:startLine,character:startCharacter}},
                    targetSelectionRange: {start:{line:startLine,character:startCharacter},end:{line:startLine,character:startCharacter}}
                }
            });        
        }
        visit(fs.readFileSync(fileName, 'utf-8'), {
            onObjectBegin: onObjectBegin,
            onObjectEnd: onObjectEnd,
            onArrayBegin: onArrayBegin,
            onArrayEnd: onArrayEnd,
            onObjectProperty: onObjectProperty,
            onLiteralValue:onLiteralValue
        });

        const m = new Map<string, ExternalJsonModelData>();
        elements.forEach(v=>{
            m.set(v.fqn, v);
        });
        this.jsonFileData.set(url, m);
        return elements;
    }
}

[msujew]

Is it a "good" way to implement external links in such a manner? Is it likely to break in future langium releases?

Given that we probably won't do any large breaking changes in regards to the scoping area, I think that's fine.

Is the implementation of HelloJsonDefinitionProvider somewhat "ok"? It seems to me, that the solution is limited to simple rule layouts: in collectLocationLinks (see above), if super.collectLocationLinks return undefined and we have a JsonReference then build the list of locations.

You could override the DefinitionProvider, though I would not recommend it. The DefaultDefinitionProvider among other services (such as the ReferencesProvider and HighlightProvider) will use the References service to perform their duty. Overriding this service is preferred, since its the source for many other LSP services.

Is the implementation of HelloJsonCompletionProvider somewhat "ok"? It seems to me, that the solution is limited to simple rule layouts: in completionFor (see above), if we have a JsonReferenceand next.feature.$containerProperty==='elements' (result of trial-and-error), then build the list of completion values.

Yeah, that seems pretty reasonable 👍

[goto40]

This may be related in #712 in some aspects (code completion)

[goto40]

I created a demo to make it easier to demonstrate what I mean.

@cdietrich I would be very happy if you could have a look. Maybe you have some better solution to the problem?

@msujew @spoenemann I think the capability to reference arbitrary external models (langium models or non-langium models) would make langium very flexible. What do you think? Comments are very welcome.

[cdietrich]

hi, i dont know when i will find time to have a look.

[spoenemann]

I would try again with your initial approach of proper cross-references before you start implementing all LSP services again. What you need to do is to ensure that the data structure you create in loadJsonFile is a proper Langium document:

• All nodes must be contained in a tree structure with a root
• Every node needs a proper $type value
• Every node except the root needs $container and $containerProperty values – this can be done easily with

  streamAst(root).forEach(linkContentToContainer);

• The root must point to a valid LangiumDocument with its $document – we have utility for that as well:

  services.shared.workspace.LangiumDocumentFactory.fromModel(root, uri);
  

• If you want services like Go to Definition to work as well, you need to generate $cstNode objects as well. I haven't tried that before, so I can't tell much about the expected effort.

[zrwsmd]

i encounter similar issue too(#1518 ),write a class extends DefaultCompletionProvider truly have a hint for outer call,

but still can not resolve reference relationship

[zrwsmd]

i guess it mainly because outer model don't have document to related to,actually ,i am considering
save outer model json data into cache and transform it into langiumDocument,can langium support read cache Document?
const uri = vscode.Uri.parse("memfs:/library.json");
vscode.workspace.fs.writeFile(uri, Buffer.from(data, "utf8"));
//root come from parse json into a root astnode
let langiumDocument=shared.workspace.LangiumDocumentFactory.fromModel(root, uri);

[msujew]

You cannot use the vscode API inside of the language server, as the language server is not actually a part of the vscode extension host process. Aside from that, yes, you can use LangiumDocumentFactory.fromModel, assuming that the root is actually a valid AST.

Note that I would still recommend you to serialize the json document into an AST and use that as a in-memory document.

[zrwsmd]

it worked!thanks for your remind You cannot use the vscode API inside of the language server and serialize the json document into an AST,code can be refered as below

stSerializer.ts(configure it in st-module.ts)
serializer: {
JsonSerializer: services => new StSerializer(services)
}

export class StSerializer extends DefaultJsonSerializer {
    constructor(services: LangiumCoreServices) {
        super(services);
    }

    override serialize(node: AstNode, options?: JsonSerializeOptions | undefined): string {
        return super.serialize(node, options);
    }

   override deserialize<T extends AstNode = AstNode>(content: string, options: JsonDeserializeOptions = {}): T {
        let jsonArrObj = JSON.parse(content);
        let composeNodeArr = jsonArrObj as ComposeNode[];
        //transform2AstNode method transform json to root astNode,this method is seen below
        let rootAstNode= transform2AstNode(composeNodeArr);
        return rootAstNode as unknown as T;
    }
}

util.ts

//ComposeNode is my custom json Obj root
export function transform2AstNode(composeNodeArr: ComposeNode[]): St {
    let root: St = {
        $type: 'St',
        function_block: [],
        itface: [],
        program: [],
        st_function: [],
        types_alias: [],
        types_enum: [],
        types_struct: [],
        types_union: []
    };
    // let astNode: StructsList | FunctionBlock | StFunction;
    composeNodeArr.forEach(composeNode => {
        let element = composeNode.elements[0];
        let elementType = element.elementType;
        if (elementType === 'functionBlock') {
            element = element as FunctionBlockElement;
            let varDeclArr = element.varDecls;
            let astNode: FunctionBlock = {
                $container: root,
                $type: 'FunctionBlock',
                methods: [],
                name: '',
                varInputs: [],
                varLocals: [],
                varOutputs: []
            };
            astNode.name = element.elementName;
            varDeclArr.forEach(varDecl => {
                let varName = varDecl.varName;
                let varType = varDecl.varType;
                let varGlobalType = varDecl.varGlobalType;
                let cacheTypeName: Native_Type_Name = {
                    $type: 'Native_Type_Name',
                    Cache_type_name: varType
                };
                if (varGlobalType) {
                    let langiumVarDecl: VarDeclarationInit;
                    if (varGlobalType === 'VAR_INPUT') {
                        let varInput: VarInput = {
                            $container: astNode,
                            $type: 'VarInput',
                            items: []
                        };
                        langiumVarDecl = {
                            $container: varInput,
                            $type: 'VarDeclarationInit',
                            nextVariables: [],
                            typeName: cacheTypeName,
                            variables: varName
                        };
                        varInput.items.push(langiumVarDecl);
                        astNode.varInputs.push(varInput);
                    } else if (varGlobalType === 'VAR') {
                        let varLocal: VarLocal = {
                            $container: astNode,
                            $type: 'VarLocal',
                            items: []
                        };
                        langiumVarDecl = {
                            $container: varLocal,
                            $type: 'VarDeclarationInit',
                            nextVariables: [],
                            typeName: cacheTypeName,
                            variables: varName
                        };
                        varLocal.items.push(langiumVarDecl);
                        astNode.varLocals.push(varLocal);
                    }
                }
            });
            root.function_block.push(astNode);
        }
    });
    return root;
}


 export function getRelatedElementAndLangiumDoc(cacheName: string): [FunctionBlockElement | undefined, LangiumDocument?] {
    //this data.json is your outer model as json format
    let data = readJsonFile(
        'D:\\glsp-vscode-integration-master-without-glsp-langium3.0-st-analyze-main-0528\\extension\\src\\data.json'
    );
    let serializer = st.serializer.JsonSerializer;
    let astNode = serializer.deserialize(data);
    let langiumDocument = shared.workspace.LangiumDocumentFactory.fromModel(astNode, URI.parse(`memory://cache.st`));
    let jsonArrObj = JSON.parse(data);
    let composeNodeArr = jsonArrObj as ComposeNode[];
    let cacheFunctionBlock = composeNodeArr.find(item => {
        return item.elements[0].elementName === cacheName;
    });
    if (cacheFunctionBlock) {
        let functionBlockElement = cacheFunctionBlock.elements[0] as FunctionBlockElement;
        return [functionBlockElement, langiumDocument];
    }
    return [cacheFunctionBlock, langiumDocument];
}

st-scope.ts

export class StScopeProvider extends DefaultScopeProvider {
    constructor(services: LangiumCoreServices) {
        super(services);
    }

    override getScope(context: ReferenceInfo): Scope {
        if (context.property === 'element') {
            const memberCall = context.container as MemberCall;
            const previous = memberCall.previous;
            if (!previous) {
                return super.getScope(context);
            }
          //this method refer to langium Lox project implement,you can change to your business code implement
            const previousType = inferType(previous, new Map());
            if (isCacheType(previousType)) {
                let cacheScope = this.scopeCache(previousType.literal);
                if (cacheScope) {
                    return cacheScope;
                }
            }
            //console.log(previousType);
        } else if (context.property === 'variable') {
            return super.getScope(context);
        }

        return super.getScope(context);
    }

    private scopeCache(cacheName: string): Scope | undefined {
        //cacheName=> e.g CacheFb2
        let [functionBlockElement, langiumDocument] = getRelatedElementAndLangiumDoc(cacheName);
        let cacheScope: Scope | undefined;
        if (functionBlockElement) {
            let cacheVarDecls = functionBlockElement.varDecls;
            const cacheDesc = stream(cacheVarDecls)
                .map(cacheVarDecl => {
                    //cacheVarDecl.$document=
                    return this.descriptions.createDescription(cacheVarDecl, cacheVarDecl.varName, langiumDocument);

                    // return undefined;
                })
                .nonNullable();
            cacheScope = this.createScope(cacheDesc, cacheScope);
        }

        if (cacheScope) {
            return cacheScope;
        }
    }

    protected override getGlobalScope(referenceType: string, context: ReferenceInfo): Scope {
        // the global scope contains all elements known to the language server
        const globalScope = super.getGlobalScope(referenceType, context);
        return globalScope;
    }
}

[zrwsmd]

below screenshot show its achievement