Publish roadmap and contributing doc?

[tribbloid]

I'm curious if your visions include making it a feature-complete NN training framework?

What will be the master plan? Integrating with Torch/TF/MXNet or build hardware-level compilation framework from scratch?

Also, what is the standard & code of conduct for contributions from the community?

I'm totally convinced of its capability and believe it can fit into the missing link between horizontal AI (Apache Spark/Apache Flink) and vertical AI (domain-specific hardware, Chisel/FIRRTL)

[breandan]

Right now, the project is still a research prototype, but we should be able to support CUDA with something like ND4J or JavaCPP/CUDA for GPU acceleration on the JVM. Ultimately, our goal is to move off the JVM to Kotlin/Native once it becomes more stable.

Currently focused on stabilizing our API, and improving support for differentiable programming. It should be possible to implement a feedforward MLP right now, but we have not gotten around to that yet. There are some toy examples here if you feel like looking around!

Longer term, it would be great to integrate with something like Relay IR or MLIR. Ideally, this would take the form of a Kotlin compiler plugin and would also require additional work to translate our graph. It's something we need to explore a bit further before committing.

Re: CoC and contributing. Haven't thought about this yet. Do you have any recommendations?

[tribbloid]

thanks a lot, agreed RelayIR in TVM is awesome. A sister project in DMLC is looking for a JVM based autodiff implementation:

https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=103089990 (Questions section)

Would you like to pitch them to merge with you? I'll also try to get their committers' (particularly Tianchi Chen) attention.

I'm not sure about Kotlin/Native at this time though. It appears to be self-contained and not prioritising interoperability with other LLVM libraries. In addition, all JIT compilers (JVM, LLVM, V8.js) may evolve to have similar performances to each other.

[breandan]

Sounds interesting, we are certainly fans of Tianqi's work. Would be happy to have a conversion to understand Gluon's needs a little more clearly. The landscape of autodiff/autograd for JVM appears somewhat bleak at the moment, ND4J has a prototype AD, although just for internal consumption. There is Nexus and Lantern, but unsure how well these interop with other JVM languages.

Agree that one of Kotlin's main benefits is strong JVM interoperability. From our experience developing Kotlin𝛁, we are convinced it is possible to implement AD as a JVM library, and could probably support Gluon's intended use case. I presume K/N also aims to support the same compatibility with other LLVM libraries, but agree the story is currently less complete.

[tribbloid]

Kotlin itself as an API is good enough, scala may be a "sweet-to-have" through its quirky dependent type implementation:

https://stackoverflow.com/questions/12935731/any-reason-why-scala-does-not-explicitly-support-dependent-types

I don't think an API in Java or Clojure serves any purpose. Java has 0 dependent type support which becomes exactly the downfall of ND4J SameDiff API, in fact, it is so verbose that people in Skymind.io end up not using it for most deeplearning4j components. Clojure/LISP is type unsafe out-of-the-box.

I'm not sure about Nexus and Lantern though, I like their functional approach and succinct API, however their decision to build everything from ground up and refusal in leveraging existing communities seems to be an oversight (which is why I believed merging with MXNet can be a win-win)

I'll use a few days to try to figure out the intention of Tianqi's team. BTW, have you met Dr Thorsten Scholak (https://github.com/tscholak) at ElementAI in your city? He is an active committer of hasktorch and might be interested in promoting your work.

[breandan]

If you’re interested in fully dependent types, Scala/Shapeless is probably your best bet. We’ve been able to get some of the benefits of dependent types with a few caveats. Equality checking is possible (e.g. shape checking and inference for ordinary vector/matrix arithmetic) at compile time. Type level arithmetic is possible with some difficulty, by enumerating all cases below a fixed constant. Practically speaking, we can only type check a fixed set of convolution sizes at compile time, and the general case can either be checked at runtime or dynamically. You can find more details in our recent workshop paper.

I know of Torsten, but have never met him in person. I think we‘re even in the same building. Will definitely reach out, thank you for the reminder!

[tribbloid]

At your service :) I'm 100% committed to Spark & Chisel so singleton-typed literal + recursive generic (similar to HList and dotty tuple) for shape safety seems no-brainer for me. I assume you have chosen Kotlin for other reasons, as I saw you complained about it on OpenReview (regarding THE workshop paper), and your code has a codegen that seems to use a fair amount of copy-pasting.

BUT I have no intention to refactor the core part (or argue which language is fitter), all languages are either a bit obsolete or a bit unstable or both. Kotlingrad is big enough, and I guess we can agree that its strength lies in mega-engineering instead of prototyping (for which torch is dominant). So I definitely favour the minimalistic approach you are already using

[breandan]

More than happy to accept PRs! If you can refactor our implementation and preserve the same guarantees, I'm all ears. Tried a bunch of different approaches, but perhaps there is a simpler implementation that offers compile time shape checking. The most recent prototype can be found in the Toy*Example.kt files, which use less metaprogramming and are somewhat easier to grok. Have experimented with other features like type safe currying, but unless we can leverage genericity somehow (bona fide union/intersection types would be great), the complexity is probably too high.

There are just four different operators you must implement to demonstrate a working implementation of AD: +, *, invoke and diff. Everything else is just syntactic sugar. The way I figure, the simpler the language, the fewer room for errors during graph lowering. Definitely still areas for improvement. I have not been able to fully generify multidimensional arrays, i.e. it would be nice to have a recursive type like Vec<Vec<Vec<...>, D2>, D1>. As it is, we need to implement a new tower (e.g. Mat, Cub) for each rank of tensor with corresponding conversions (it's a real headache). Given the semantics of type erasure on the JVM, I'm not sure if this is possible but would be happy to be proven wrong.

[tribbloid]

I'm still observing: https://discuss.mxnet.io/t/what-is-the-status-of-gluon-api-for-scala-clojure-jvm-autograd-api/5088

Did you get response from Thorsten?

[breandan]

Yeah, I emailed Torsten at your suggestion and he ended up giving a nice talk about his work on Hasktorch at our reading group yesterday. Only wish I had reached out sooner, thanks @tscholak!

Also had a chat with my labmate and TVM contributor @silentspring2 the other day. Sounds like it would be a great direction to explore, happy to continue discussing either here or on your thread.

[tribbloid]

That's a lot of progress, I'm in Toronto so kind of missing all the action there ...

I'll seek to post a MXNet JIRA ticket and keep our conversation there. Hope your committer friend won't mind

[tscholak]

lol, you guys know each other? small world!
hope you’re doing alright @tribbloid

[tribbloid]

@tscholak no we don't which is totally my miscalculation.
I assume you are both contrarians there (amongst colleagues who are totally comfortable with Python 2), hope this can change soon

[breandan]

From what I can tell, there are a few paths we could take to support GPUs. In no particular order:

• There were a bunch of OpenJDK projects circa 2017, including Project Trinity, Project Sumatra, Project Panama, and Project Valhalla the last two of which appear to be somewhat active.

• There is AMD's AparAPI, which appears to have been abandoned in 2016/2017.

• There is Intel's oneAPI, which has a Java API, documentation including some form of GPU support and is active, but appears to target a higher level set of use cases.

• There is Rootbeer, a Java GPU compiler, which looks interesting but appears to have died.

• A lot of JVM activity appears to have shifted to Truffle/GraalVM and for our use case GrCUDA looks very promising, has the best documentation and supports polyglot programming.

• Alternatively, the ONNX Runtime now has a Java API, which provides GPU support for running ONNX models, which would give us access to the GPU by lowering to the ONNX IR.

• Finally, there is @jcuda, JOCL and JavaCPP which have discussed merging efforts.

There is also some word of upcoming JDK features (@Craigacp?) but it looks like these are our best options for hardware acceleration at the moment. Any other suggestions welcome!