diff --git a/proposals/script-definition-template.md b/proposals/script-definition-template.md deleted file mode 100644 index c0397f292..000000000 --- a/proposals/script-definition-template.md +++ /dev/null @@ -1,129 +0,0 @@ -# Script Definition Template - -Goal: flexibly defining script behaviour using Kotlin syntax and simple usage scheme. - -## Feedback - -Discussion of this proposal is held in [this issue](https://github.com/Kotlin/KEEP/issues/28) - -## Use cases - -* Build scripts (Gradle/Kobalt) -* Test scripts (Spek) -* Command-line utilities -* Routing script for ktor (get(“/hello”) {...} on top level) -* Type-safe configuration files -* In-process scripting for IDE -* Consoles like IPython Notebook - -## Proposal - -Script definition templates are written as a regular class optionally annotated with specific annotations. For example: - -``` -@ScriptTemplateDefinition( - resolver = GradleDependenciesResolver::class, // optional, could be used for additional dependencies resolution - scriptFilePattern = "build.gradle.kts" // optional, provides a pattern for script discovery in IDE and compiler -) -open class GradleScript(project: Project, val name: String) : Project by project { - fun doSomething() { println(name) } -} -``` - -This class becomes a base class for generated script class. - -Parameters of the primary constructor of the class describe the parameters of the script; varargs and default parameter values are allowed. Regular parameters (non-fields) become hidden in script body (e.g. in the example above `project` parameter is not accessible in the script body.) - -Base dependencies of the script are equal to the dependencies of the template class itself, provided via usage mechanism (see below). Additional dependencies and implicit imports are extracted by a classes passed to `@ScriptDependencyResolver` annotation, which should be then supplied along with template class. - -## Usage - -To use “templated” scripts in the compiler/IDE, the means to recognising and handling these scripts should be provided. It is enough to provide compiler/IDE with the fully qualified name of the class with a classpath containing all dependencies of that template class and resolvers specified in the `@ScriptTemplateDefinition` annotation. Alternatively the template class could be searched in the classpath by the `@ScriptTemplateDefinition` annotation. - -In the user-controlled execution environment, e.g. in gradle, that could be achieved by creating special kind of `KotlinScriptDefinition` and passing it to the compiler, along with properly constructed `ClassLoader`. -In the IDE this could be implemented using the specific extension point: - -``` -interface ScriptTemplateProvider { - val id: String // for resolving ambiguities (together with version field) - val version: Int - - val isValid: Boolean // to simplify implementation of dynamic discovery - - val templateClassName: String - val dependenciesClasspath: Iterable - - val environment: Map? // see Dependencies section for explanation -} -``` - - -From the command line a parameter could be used to specify a template class name, and regular compilation classpath could be used for dependencies. - -Additionally, the automatic discovery of the templates marked by `@ScriptTemplateDefinition` annotation could be used with libraries that do not have plugins able to provide an extension. This could for example be used for test frameworks. - -## Script Files - -To find the script definition corresponding to the script file, the compiler uses one of the following methods: - -* Use the script definition specified explicitly in CompilerConfiguration (the Gradle case); -* Scan for all .jar in the classpath, load the list of script template definitions in each .jar from the JAR metadata, and detect the applicable one based on the `@ScriptFilePattern` annotation; -* Use an explicit annotation referring to the FQ name of the script definition class: - -``` -@file:ScriptTemplate("org.jetbrains.kotlin.gradle.GradleScript") -``` - -## Dependencies - -Dependencies required by the script are provided by the resolvers specified in parameters to `@ScriptTemplateDefinition` annotation on the template class. These are expected to implement the following interface: - -``` -interface KotlinScriptDependenciesResolver { - @AcceptedAnnotations(...) // allows to specify particular types of annotations accepted by the resolver - fun resolve(script: ScriptContents, - environment: Map?, - previousDependencies: KotlinScriptExternalDependencies? - ): KotlinScriptExternalDependencies? -} -``` - -The method is called after script parsing in compiler and IDE and allows resolver to discover particular script dependencies using any annotations as well as to pass predefined dependencies for all scripts built with appropriate template. The parameters: - -* script - the interface to the script file being processed defined as - ``` - interface ScriptContents { - val file: File? - val annotations: Iterable - val text: CharSequence? - } - ``` - where: - * file - script file, if it is a file-based script - * annotations - a list of file-targeted annotations from the script file filtered according to `AcceptedAnnotations` annotation - * text - an interface to the script contents -* environment - a map of entries representing environment specific for particular script template. The environment allows generally stateless resolver to extract dependencies according to the environment. E.g. for the gradle it could contain the gradle's `ProjectConnection` object used in the gradle IDEA plugin, allowing to reuse the project model already loaded into the plugin. The values are taken from the `ScriptTemplateProvider` extension point or script compilation call parameter. Could also contain a predefined set of parameters, e.g. “projectRootPath” -* previousDependencies - a value returned from the previous call to resolver, if any. It allows generally stateless resolver to implement an effective change detection logic, if the resolving is expensive - -Returned `KotlinScriptExternalDependencies` is defined as: - -``` -interface KotlinScriptExternalDependencies { - val javaHome: String? = null // JAVA_HOME path to use with the script - val classpath: Iterable get() = emptyList() // dependencies classpath - val imports: Iterable get() = emptyList() // implicit imports - val sources: Iterable get() = emptyList() // dependencies sources for source navigation in IDE - val scripts: Iterable get() = emptyList() // additional scripts to compile along with the current one -} -``` - -This schema allows user to implement any annotation-based syntax for dependency resolution, and if it is not enough, perform any parsing of the file directly. - -## Further ideas to consider - -* `ScriptTemplateProvider` extension point could be extended to provide dependencies changes notification mechanism to an IDE. That could help for example then dependencies are defined in a file other that script itself, so there is no way for IDE to detect the right moment to ask about changed dependencies. -* Additional possible annotations (or main annotation parameters) on the template class: - * `ScriptDependencies` - for defining simple dependencies like JDK, kotlin stdlib or files with simple file searching scheme (e.g. from project's lib folder) - * `ScriptDependenciesRepository` - for using with a “standard” dependency resolver, like maven, could be used together with a form of `ScriptDependencies` annotation accepting library coordinates. - * `ScriptImplicitImports` - as a compliment to the direct dependencies specification, to allow specifying implicit imports directly - diff --git a/proposals/scripting-support.md b/proposals/scripting-support.md new file mode 100644 index 000000000..242ec4c5c --- /dev/null +++ b/proposals/scripting-support.md @@ -0,0 +1,780 @@ + +# Kotlin Scripting support + +*Replaces [Script Definition Template](https://github.com/Kotlin/KEEP/blob/master/proposals/script-definition-template.md) +proposal.* + +## Feedback + +Discussion of this proposal is held in [this issue](https://github.com/Kotlin/KEEP/issues/75) + +## Motivation + +- Define Kotlin scripting and it's applications +- Describe intended use cases for the Kotlin scripting +- Define scripting support that is: + - applicable to all Kotlin platforms + - provides sufficient control of interpretation and execution of scripts + - simple enough to configure and customize + - provides usable default components and configurations for the typical use cases +- Provide basic examples of the scripting usage and implementation +- Address the issues found during the public usage of the current scripting support + +## Status of this document + +The document is still a draft, and few important parts are still missing, in particular: + +- REPL: + - API for REPL host configuration and embedding + - API for REPL plugins (maybe should be covered elsewhere): + - custom repl commands + - highlighting + - completion + - etc. +- IDE plugins API for custom scripting support (besides discovery) +- ? + +## Table of contents + +- [Applications](#applications) +- [Basic definitions](#basic-definitions) +- [Use cases](#use-cases) + - [Embedded scripting](#embedded-scripting) + - [Standalone scripting](#standalone-scripting) + - [Project infrastructure scripting](#project-infrastructure-scripting) + - [Script-based DSL](#script-based-dsl) +- [Proposal](#proposal) + - [Architecture](#architecture) + - [Script definition](#script-definition) + - [Script Compilation](#script-compilation) + - [Script Evaluation](#script-evaluation) + - [Standard hosts and discovery](#standard-hosts-and-discovery) + - [How to implement scripting support](#how-to-implement-scripting-support) + - [Implementation status](#implementation-status) + - [Examples](#examples) + +## Applications + +- Build scripts (Gradle/Kobalt) +- Test scripts (Spek) +- Command-line utilities +- Routing scripts (ktor) +- Type-safe configuration files (TeamCity) +- In-process scripting and REPL for IDE +- Consoles like IPython/Jupyter Notebook +- Game scripting engines +- ... + +## Basic definitions + +- **Script** - a text file written in Kotlin language but allowing top-level statements and having access to some + implicit (not directly mentioned in the script text) properties, functions and objects, as if the whole script body + is a body of an implicit function placed in some environment (see below) +- **Scripting Host** - an application or a component which handles script execution +- **Scripting Host Environment** - a set of parameters that defines an environment for all scripting host services, + contains when relevant: project paths, jdk path, etc. It is passed on constructing the services +- **REPL statement** - a group of script text lines, executed in a single REPL eval call +- **Script compilation configuration** - a set of parameters configuring a script compilation, such as dependencies, + external (global) variables, script parameters, etc. +- **Script definition** - a set of parameters and services defining a script type +- **Compiled script** - a binary compiled code of the script, stored in memory or on disk, which could be loaded + and instantiated by appropriate platform +- **Dependency** - an external library or another project whose declarations are available for the script being compiled + and evaluated +- **Imported script** - another script whose declarations are available for the script being compiled and evaluated +- **Execution environment** - the environment in which the script is executed, defining which services, objects, + actions, etc. are accessible for the script + +## Use cases + +### Embedded scripting + +The use case when a scripting host is embedded into user's application, e.g. specialized console like +IPython/Jupyter notebook, Spark shell, embedded game scripting, IDE and other application-level scripting. + +#### Environment and customization + +In this case the script is most likely need to run in a specific execution environment, defined by the scripting host. +The default script compilation and evaluation configurations are defined by the scripting host as well. The host may +provide script authors with a possibility to customize some configuration parameters, e.g. to add dependencies or +specify additional compilation options, e.g. using annotations in the script text: + +``` +@file:dependsOn("maven:artifact:1.0", "imported.package.*") +@file:require("path/to/externalScript.kts") +@file:compilerOptions("-someCompilerOpt") +``` + +If the host need to support several script *types*, i.e. sets of compilation configurations and customization means, +there should be a way for the host to distinguish the scripts and select appropriate set of +compilation/configuration/evaluation services and properties. The host authors can implement the selection based on any +script property, but due to the difficulties in supporting file type distinction based on anything but filename +extension across platforms and IDEs, the default implementations should support only extension-based selection. + +*Note: It would be nice to provide an infrastructure (complete hosts or libraries) that support some typical mean for +each platform to resolve external libraries from online repositories (e.g. - maven for JVM) out of the box.* + +#### Typical usages + +In a simple case, the developer wants to implement a scripting host to control script execution and provide the required +environment. One may want to write something as simple as: +``` +KotlinScriptingHost().eval(File("path/to/script.kts")) +``` +or +``` +KotlinScriptingHost().eval("println(\"Hello from script!\")") +``` +and the script should be executed in the current environment with some reasonable default compilation and evaluation +settings. If things need to be configured explicitly, the code would look like: +``` +val scriptingHost = KotlinScriptingHost(configurationParams...) +scriptingHost.eval(File("path/to/script.kts")) +``` +This would also allows the developer to control the lifetime of the scripting host. + +In some specific cases it is desired to perform additional action between compilation and actual evaluation of the +script, e.g. verify the compiled script. In this case, the usage may look like: +``` +val scriptingHost = KotlinScriptingHost(configurationParams...) +val compiledScript = scriptingHost.compile(File("path/to/script.kts")) +// do some verification, and if it succeeds +scriptingHost(compiledScript, evaluationParams...) +``` +This snipped provides also an example of passing specific parameters to the evaluation, which should be supported for +the simpler usages as well. +And also in this form the compiled script could be evaluated more than once. + +#### More control + +To be able to run scripts in a user controlled environment, the following information bits could be configured or +provided to the host: +- *Script Compiler* - the service that will compile scripts into a form accepted by the Evaluator +- *Script Compilation Configuration* - set of properties defining the script compilation settings, including: + - *Script Base Class* - an interface (or prototype) of the script class, expected by the executor, so the compiler + should compile script into the appropriate class + - *Dependencies* - external libraries that could be used in the script + - *Default imports* - import statements implicitly added to any compiled script + - *Environment variables* - global variables with types that is assumed visible in the script scope + - etc. +- *Script Compilation Configurator* - the service that provides a dynamic part of the configuration for a script + compilation, so the part dependent on the compiled script content. +- *Script Evaluator* - the service that will actually evaluate the compiled scripts in the required execution + environment +- *Evaluation environment* - set of properties defining an environment for the evaluation, including: + - *Environment variables* - actual values of the environment variables from compilation configuration + - etc. + +Some of these parameters could be wrapped into a *Script Definition* for easier identification of the script types +by the hosts that may support handling of the several script types simultaneously. + +#### Caching + +Since calling Kotlin compiler could be quite a heavy operation in comparison with typical script execution, the caching +of the compiled script should be supported by the compilation platform. + +#### Execution lifecycle + +The script is executed according to the following scheme: +- compilation - the *Script Compiler* takes the script, it's configurator and/or configuration and + provides a compiled class. Inside this process, the following activity is possible: + - configuration refinement - if the compilation configured accordingly, the configurator is called after parsing to + refine the configuration taking into account the script contents: +- evaluation - the *Evaluator* takes the compiled script instantiates it if needed, and calls the appropriate method, + passing arguments from the environment to it; this step could be repeated many times + +#### Processing in an IDE + +The IDE support for the scripts should be based on the same *Script definition*. Basically after recognizing the script +*type* (see *Environment and customization* section above), the IDE may use the *Compilation Configuration* and use its +parameters to implement highlighting, navigation and other features. The default implementation of Kotlin IDEA plugin w +should support the appropriate functionality, based on the standard set of configuration parameters. + +### Standalone scripting + +Standalone scripting applications include command-line utilities and a standalone Kotlin REPL. + +Standalone scripting is a variant of the embedded scripting with hosts provided with the Kotlin distribution. + +#### Hosts + +The standalone script could be executed e.g. using command line Kotlin compiler: + +`kotlinc -script myscript.kts` + +Or with the dedicated runner included into the distribution: + +`kotlin myscrtipt.kts` + +To be able to use the Kotlin scripts in a Unix shell environment, the *shebang* (`#!`) syntax should be supported +at the beginning of the script: +``` +#! /path/to/kotlin/script/runner -some -params +``` + +*Note: due to lack of clear specification, passing parameters in the shebang line could be +[problematic](https://stackoverflow.com/questions/4303128/how-to-use-multiple-arguments-with-a-shebang-i-e/4304187#4304187), +therefore alternative schemes of configuring scripts should be available.* + +#### Script customizations + +It should be possible to process custom scripts with the standard hosts, e.g. by supplying a custom script definition +in the command line, e.g.: + +`kotlin -scriptDefinition="org.acme.MyScriptDef" -scriptDefinitionClasspath=myScriptDefLib.jar myscript.kts` + +In this case the host loads specified definition class, and extract required definition from it and its annotations. + +Another possible mechanism is automatic discovery, with the simplified usage: + +`kotlin -cp=myScriptDefLib.jar myscript.myscr.kts` + +In this case the host analyses the classpath, discovers script definitions located there and then processes then as +before. Note that in this case it is should be recommended to use dedicated script extension (`myscr.kts`) in every +definition to minimize chances of clashes if several script definitions will appear in the classpath. And on top of +that, some clash-resolving mechanism is needed. + +#### Script parameters + +For the command line usage the support for script parameters is needed. The simplest form is to assume that the script +has access to the `args: Array` property/parameter. More advanced is to have a customization that supports a +declaration of the typed parameters in the script annotations e.g.: + +``` +@file:param("name", "String?") // note: stringified types are used for the cases not supported by class literals +@file:param("num", Int::class) +@file:param("list", "List") + +// this script could be called with args "-name=abc -num=42 -list=a,b,c" +// and then in the body we can access parsed typed arguments + +println("${name ?: ""} ${num/6}: ${list.map { it.toUpperCase() } }") +``` + +#### IDE support + +Since in this use case scripts are not part of any project, and content-based script type detection appears quite +difficult, the possibility of reasonable IDE support is questionable. + +#### Standalone REPL + +Standalone REPL is invoked by a dedicated host the same way as for standalone script but accepts user's input as repl +statements. It means that the declarations made in the previous statements are accessible in the subsequent ones. +In this mode, all new scripting features should be accessible as well, including customization. + +### Project infrastructure scripting + +Applications: project-level REPL, build scripts, source generation scripts, etc. + +Project infrastructure scripts are executed by some dedicated scripting host usually embedded into the project build +system. So it is a variant of the embedded scripting with the host and the IDE support integrated into build system +and/or IDE itself. + +#### IDE support + +From an IDE point of view, they are project-context dependent, but may not be part of the project sources. (In the same +sense as e.g. gradle build scripts source is not considered as a part of the project sources.) + +#### Discovery + +The IDE needs to be able to extract scripts environment configurations from the project settings. + +#### Project-level REPL + +A REPL that has access to the project's compiled classes. + +### Script-based DSL + +Applications: test definition scripts (Spek), routing scripts (ktor), type safe config files, etc. + +In these cases the scripts are considered parts of the kotlin project and are compiled to appropriate binary form by the +compiler, and then linked with the rest of the compilation results. They differ from the other project's sources by the +possibility to employ script semantic and configurability and therefore avoid some boilerplate and make the sources +look more DSL-like. + +In this scenario, no dedicated scripting host is used, but the standard compiler is used during the regular compilation +according to configured script recognition logic (e.g. the script type discovery mechanism described above), and the +target application should implement its own logic for instantiating and calling the generated script classes. +Script compiler may also annotate generated classes and methods with user-specified annotations to integrate it with +existing execution logic. E.g. junit test scripts could be annotated accordingly. + +From an IDE point of view, these scripts are the part of the project but should be configured according to the +recognized script definition. + +## Proposal + +### Architecture + +#### Components + +The scripting support consists of the following components: +- **ScriptCompiler** - interface for script compilation + - compilation: `(scriptSource, compilationConfigurator, additionalCompilationConfiguration) -> compiledScript` + - predefined script compilers based on the kotlin platforms: /JVM, /JS, /Native + - custom/customized implementation possible + - compiled scripts cashing belongs here + - should not keep the state of the script compilation, the required state for the subsequent compilations, e.g. in the + REPL mode, is passed along with the compiled script +- **ScriptCompilationConfigurator** - provides *static* and content-dependent configuration properties for the compiler + - some basic implementations are provided, but for custom scripting user may provide an implementation + - `defaultConfiguration` property of type `ScriptCompileConfiguration` + - `refineConfiguration(scriptSource, configuration, processedScriptData) -> ScriptCompileConfiguration` + - the `refineConfiguration` configuration is called only if the *static* configuration has specific *refine request* + properties, and the passed `processedScriptData` contains appropriate data extracted from parsing. E.g. if the + *static* configuration contains a property `refineConfigurationOnAnnotations`, the `refineConfiguration` will be + called after parsing and the `processedScriptData` parameter will contain a list of parsed annotations. +- **ScriptEvaluator** - the component that receives compiled script instantiates it, and then evaluates it in a required + environment, supplying any arguments that the script requires: + - evaluation: `(compiledScript, environment) -> Any?` + - the `compiledScript` contains the final compilation configuration used + - the `environment` is an entity denoting/referencing the actual execution environment of the script + - predefined platform-specific executors available, but could be provided by the scripting host + - possible executors + - JSR-223 + - IDEA REPL + - Jupyter + - Gradle + - with specific coroutines context + - ... +- **IDE support** - Kotlin IDEA plugin should have support for scripting with script definition selection based on the + file name extension, and also includes discovery. The exposed generic ide support that would allow to build rich + script editing apps and REPLs is outside of the scope of this proposal and will be covered elsewhere. + +#### Data structures + +- `ScriptSource` determines the way to access script for other components; it consists of: + - the script reference pointer: url + - an accessor to the script text + Both components are optional but at least one is required for a regular script. + The class implements source position and fragment referencing classes used e.g. in error reporting. +- `KotlinType` a wrapper around Kotlin types, used to decouple script definition and compilation/evaluation + environments. It could be constructed either from reflected or from stringified type representation. +- `ChainedPropertyBag` - a heterogeneous container for typed properties, that could be chained with a "parent" + container, which is being searched for a property, if it is not found in the current one. This allow to implement + properties overriding mechanisms without copying, e.g. for configuration refining. Used for all relevant properties, + like compilation configuration and evaluation environment. +- `ScriptDefinition` - a facade combining services and properties defining a script type. Could be constructed manually + or from annotated script base class. + +### Script definition + +Script Definition is a facade combining services and properties in one entity for simplified discovery and +configuration of a script type. It also defines the base class for the generated script classes (in the properties), +and therefore defines the script "skeleton" and DSL. +``` +interface ScriptDefinition { + + // constructor(environment: ScriptingEnvironment) // the constructor is expected from implementations + + val properties: ScriptDefinitionPropertiesBag + + val compilationConfigurator: ScriptCompilationConfigurator + + val evaluator: ScriptEvaluator<*>? +} +``` + +#### Declaring script definition + +The definition could be constructed manually, but the most convenient way, which is also supported by the discovery +mechanism, is to annotate the appropriate script base class/interface with the script defining annotations and then +allow provided scripting infrastructure to construct the definition from it. For example: + +``` +@KotlinScript("My script") +@KotlinScriptFileExtension("myscr.kts") +@KotlinScriptCompilationConfigurator(MyConfigurator::class) +@KotlinScriptEvaluator(MyEvaluator::class) +abstract class MyScript(project: Project, val name: String) { + fun helper1() { ... } + + [@ScriptBody] + [suspend] abstract fun (params...): R +} +``` + +*Where:* +- any valid method name could be used in place of `` +- `@ScriptBody` annotation marks the method the script body will be compiled into. In the absence of the explicit + annotation the SAM notation will be used +- `interface` or `open class` could be used in place of the `abstract class` + +The annotations have reasonable defaults, so in the minimal case it is enough to mark the class only with the +`@KoltinScript` without parameters. But it is recommended to give a dedicated file name extension for every script +type to minimize chances for clashes in case of multiple definitions in one context. + +#### Static compilation configuration + +In cases then script compilation configuration is static, instead of defining custom `ScriptCompilationConfigurator` +it is possible to declare configuration statically in an object or class implementing the `List` interface holding +configuration properties, and use appropriate annotation that will instruct the scripting infrastructure to create +a configurator from these properties. E.g.: + +``` +object MyScriptConfiguration : List, Any?>> by ArrayList, Any?>>( + listOf( + ScriptCompileConfigurationProperties.defaultImports to listOf("java.io.*") + ) +) + +@KotlinScript +@KotlinScriptFileExtension("myscr.kts") +@KotlinScriptDefaultCompilationConfiguration(MyScriptConfiguration::class) +interface MyScript +``` + +*(see possible compilation configuration properties below)* + +### Script Compilation + +#### Script Compiler + +Script compiler implements the following interface: +``` +interface ScriptCompiler { + + suspend fun compile( + script: ScriptSource, + configurator: ScriptCompilationConfigurator? = null, + additionalConfiguration: ScriptCompileConfiguration? = null // overrides parameters from configurator.defaultConfiguration + ): ResultWithDiagnostics> +} +``` +The compilers for the supported platforms are supplied by default scripting infrastructure. + +#### Script Compilation Configurator + +The Script Compilation Configurator is a class implemented by the script definition author, or provided by the +infrastructure from the author-supplied static configuration properties. It should implement the following interface: +``` +interface ScriptCompilationConfigurator { + + // constructor(environment: ScriptingEnvironment) // the constructor is expected from implementations + + val defaultConfiguration: ScriptCompileConfiguration + + suspend fun refineConfiguration( + scriptSource: ScriptSource, + configuration: ScriptCompileConfiguration, + processedScriptData: ProcessedScriptData = ProcessedScriptData() + ): ResultWithDiagnostics = + defaultConfiguration.asSuccess() +} +``` + +#### Compilation Configuration Properties + +The following properties are recognized by the compiler: +- `sourceFragments` - script fragments compile - allows to compile script partially +- `scriptBodyTarget` - defines whether script body will be compiled into resulting class constructor or to a + method body. In the latter case, there should be either single abstract method defined in the script base class, or + single appropriate method should be annotated accordingly (*TODO:* elaborate) +- `scriptImplicitReceivers` - a list of script types that is assumed to be implicit receivers for the script body, as + if the script is wrapped into `with` statements, in the order from outer to inner scope, i.e.: + ``` + with(receivers[0]) { + ... + with(receivers.last()) { +