proposal: create a JS async language plugin to play with top-level-ish await

[hamilton]

Working iteratively with async javascript is not always very pretty if you're coming from a data science background. We've built a number of workarounds that help:

1.) fetch chunks make it simple to load data from the web + from the iodide server, which removes the programmatic headaches of working with async js
2.) if your return value of a chunk is a Promise, iodide awaits until the Promise resolves before moving onto the next enqueued chunk

But there's probably more we can do, or at least experiment with. These aren't particularly natural ways of handling data pulling in all situations.

If #1840 lands, we should be able to better parametrize language chunks and try out new ideas. One idea would be to create an async js plugin, which actually just wraps the chunk contents in an async Function, runs it, and then puts whatever is the return value into window[returnVarName] (of which returnVarName is defined as a flag).

This would allow some workflows that are currently difficult boilerplatey with vanilla js chunks:

%% async-js returnVarName="crunchedData"

// even though we use var here, it will be scoped to the chunk only because of how 
// Function works.
var data = await fetchMyData() 

var moreData = await combineTheseDataSources(data, etc.);

moreData // this is the return value added to window.crunchedData
%% js

// plot crunchedData

plot(crunchedData, ...)

I once the flags-in-evaluators issue gets fixed, this might be the cheapest way to use a language plugin to explore this idea. If we like it and are okay with the tradeoffs, and feel that it is valuable enough for Iodide users, we might explore just adding an async flag into iodide js chunks that replicates this behavior.

The tradeoffs primarily are around scoping – var doesn't assign globally as it does elsewhere, nor would something like x = await fetch().... This is a bit of a dealbreaker for casual data science, I think, sine users are still expecting the REPL experience. Unless something speeds up this TC39 proposal I don't know any way around this.

[bcolloran]

thinking about this a little, and even thought it contradicts much of what i say in #1840, a natural approach would be to have e.g:

%% js-async globals=data1,data2,data3
data1 = await fetchText("https://vincentarelbundock.github.io/Rdatasets/csv/datasets/iris.csv")
data3 = await fetchJson("https://www.anapioficeandfire.com/api/characters?page=1&pageSize=10")
data3 = await fetchJson("https://www.anapioficeandfire.com/api/characters?page=2&pageSize=10")

be nothing more than sugar for:

%% js
var data1,data2,data3
(async ()=>{
  data1 = await fetchText("https://vincentarelbundock.github.io/Rdatasets/csv/datasets/iris.csv")
  data3 = await fetchJson("https://www.anapioficeandfire.com/api/characters?page=1&pageSize=10")
  data3 = await fetchJson("https://www.anapioficeandfire.com/api/characters?page=2&pageSize=10")
})()

see https://alpha.iodide.io/notebooks/2327/

as in that NB, this kind of immediately-invoked async function returns a promise, and thus blocks the eval queue automatically. but also, since it returns a promise unlike an eval the final statement is not returned... tho a user could add a return to get a value in the console.

but i don't know... for something that desugars so simply to a pattern that is pretty clean and interacts pretty nicely with our existing evaluation queue... i wonder if this is the kind of thing where it is better to document this pattern/snippet than add sugar for it. (@hamilton and i had a long discussion today about what provides the most long-term legibility and usability... one of the difficult tradeoffs we discussed is how much magic sugar to add for stuff like this. it can make it nice for power users, but when all the power user early adopters use it everywhere, it can become inscrutable for novices)

[dmose]

The main use case I have at this moment is awaiting fetches from the Bugzilla REST API. As things stand, it just means that I have to nest things in functions that I might ideally do at the top-level. It's not a big deal.

That said, I'm coming to the conclusion that one of the hardest things about Iodide is the unusual scoping environment, and I'd rather not see that get worse.

[bcolloran]

oh duh, also my simple proposal would not work at all if the user declares their variables. e.g.:

%% js-async globals=data1,data2,data3
var data1 = await fetchText("https://vincentarelbundock.github.io/Rdatasets/csv/datasets/iris.csv")
var data3 = await fetchJson("https://www.anapioficeandfire.com/api/characters?page=1&pageSize=10")
var data3 = await fetchJson("https://www.anapioficeandfire.com/api/characters?page=2&pageSize=10")

mapping to

%% js
var data1,data2,data3
(async ()=>{
  var data1 = await fetchText("https://vincentarelbundock.github.io/Rdatasets/csv/datasets/iris.csv")
  var data3 = await fetchJson("https://www.anapioficeandfire.com/api/characters?page=1&pageSize=10")
  var data3 = await fetchJson("https://www.anapioficeandfire.com/api/characters?page=2&pageSize=10")
})()

would mean that the vars declared in the inner scope clobber the vars declared in the outer global scope. :-(

hmmm as i think about this more, i'm increasingly inclined to think that it's probably is the kind of thing that looks simple but is not, and that introduces more special case confusion than it fixes :-(

[dmose]

Agreed re the confusion on this implementation front.

Here's a weird question: any chance that somehow wrapping the md code in an ES6 module before evalling it would solve any of the existing or propose scope weirdness?

[bcolloran]

hmm, off the top of my head, i don't think they should interact. the Markdown stuff (and other code chunks) are fairly orthogonal to the JS eval scope.

[dmose]

I think I misspoke. I wasn't really referring to MD stuff specifically, but all the JS stuff in Iodide. Which is to say, I wonder if wrapping the JS code in Modules before evalling would make the scope setup less weird... Admittedly a tangent to this issue.

[bcolloran]

to be consistent with the experience of other interactive computing tool like Jupyter, RStudio, etc, we want to make sure that code evaled by users has access to the global scope, and that variables defined at the top-level in evaled code "leak" into the global scope. so wrapping every evaluation in a module might make the semantics of let and const more consistent, but it would complicate the experience in general if you had to export/import to access stuff elsewhere in your notebook)

(or perhaps I am not understanding what you mean?)