Is it okay to use hash tables in Pallene?

[ewmailing]

I have a need to use a hash table (not a contiguous array) in Pallene. I know I could try to funnel this through a Lua function and do this on the Lua side, but the elements in my hash table are Pallene records and I have lots of little local helper functions in Pallene, so it would be more convenient to do this on the Pallene side.

I'm trying to figure out what's okay to do on the Pallene side, and what must be done on the Lua side. So far my impression (through experimentation) is:

• I must insert new entries into the hash table on the Lua side, otherwise it looks like Pallene will try to create a contiguous array. (My test case was a sparse array with random integer keys and it feels I got a very large contiguous array based on the ordering and slowness.)
• It appears I can pass next() and pairs() from Lua to Pallene and I can write for-in pairs() loops on the Pallene side and it seems like Pallene does the right thing. (I did find a semi-related compiler bug to this which I will file, but it has a work-around.)
• I'm not sure about removing entries yet (i.e. setting to nil).
• I'm also not sure yet if it is okay to create an empty table on the Pallene side, e.g. t = {} (which will have keys inserted on the Lua side).

Are there any other considerations or warnings I should be aware of?

Thanks

[hugomg]

Proper hash tables might be the most glaring omission in the Pallene type system right now. The problem, however, is that we still haven't figured out the best way to add a hash table type to the type system. In particular, how to deal with "not found". For example, consider this bit of Lua:

function greet(t, k)
    local v = t[k]
    if v then
        return "hello"..v
    else
        return "default"
    end
end

To support this kind of idiom, we would need to tell the type system that if we test v in an if statement, then inside the then branch we can assume that v is not nil. Basically, outside the if statement the v has type any while inside it has type string. Do we want to allow this sort of thing? Perhaps. I know Teal does it, which we could use for inspiration.

Alternatively we could go for a design with a simpler type system but which requires more type annotations.

local va = t[k]
if va then
    local v = va as string
    return "hello"..v
else

It's clunkier, but maybe it's a good solution to at least get some support for hash tables out of the gate?

Anyway, I'm all ears! Would love to know if yall have any suggestions about the hash tables.

[ewmailing]

I don't have strong opinions on language design. I tend to mostly go with the flow. But it seems to me that optionals are all the rage now. And from my non-language designer vantage point, it seems to me that this may be this might be the approach worth taking. Maybe at a minimum:

function greet(t:{string?}, k:integer) : string?
    local v = t[k]
    if v then
        return "hello" .. v
    else
        return nil
    end
end

Whereas if the user doesn't specify the optionals, it works like Pallene does today, and requires the object with the specified type to exist.

Then maybe more advanced, maybe Pallene could introduce more syntax and built-in behaviors to handle this type of optional nil checking or unwrapping more automatically.

As for hash-table support out of the gate, I am not in a rush for this since it can be accomplished by passing through to Lua. Honestly, I am much more interested in your work on the C FFI. In this area, I am hoping/expecting you to do things that don't have an easy analog to Lua.

My interest in Pallene is really centered on the performance aspects, while I am much more indifferent on the type system. So my only other input on what you do with this type system problem is suggest, if there is an obvious decision you can make that can lead Pallene to generating more performant code, even at the trade off of not getting sooner, I would still prefer the more performant version. After that, I probably don't have enough expertise in this area to say any more. (Although, I might have a lot more opinions on the C FFI and other related performance issues.)

[bjornbm]

@hugomg: Personally I prefer your second option (but inlining the as string to be cosmetically closer to the first option):

local v = t[k]
if v then
    return "hello " .. v as string
else

I think I prefer this over an implicit cast to string (your first option) based on the assumption that “that is clearly what the programmer meant when they used ..”. All that is known after this if is that v is neither nil nor false, not that v is a string. My feeling is that the as string is a small cost to pay to reduce the risk of me shooting myself in the foot by typing too fast and thinking about types too little. (This is related to #494 where I will also comment shortly.)

Apart from the hash table lookup itself, the code above seems to be consistent with the current type system (that is, no special handling/assumptions needed from the if statement).

Full example, working right now for arrays:

function m.greet(t : { any }, k : any) : string
    local v = t[k]
    if v then
        return "hello " .. v as string
    else
        return "default"
    end
end

Usage in lua: print(m.greet({ [1] = 'Hugo' }, 1))

With this syntax, all that is needed is actually getting hash tables to work. Then the above m.greet should still work except that the type of t probably has to change?

Also, on the topic of implicit versus explicit casting: I am not very comfortable with the k : any in the first line. I would have preferred if pallene forced me to write either k : integer or local v = t[k as integer] (the former is of course better in this case as it allows to check calls from pallene). But as written above would be “good” for a future general Lua/hash table where the key type isn't constrained.

[hugomg]

In the mean time, the (painful and slow) workaround is what you've been doing. Do the hash table manipulation in Lua and expose that to Pallene via functions.