Something I've been thinking about with expansion of library: a decent amount of the work we've been using involves application of inductive biases and teacher-prompted training to model architecture. Currently we have:
- Teacher-student forcing: lstms and transformers
- Expert curricular training: edit action transducer
- Monotonicity: hard attention lstm
- Hard alignment: also hard attention lstm
One thing I would like to do with the next overhaul is modularize these beyond their respective models (like we're trying to do with #77 for teacher forcing) so that they can be 'dropped in' wherever. This would allow 'fun' combinations such as:
- Feature-invariant transformer with monotonic assumptions and hard alignment
- Hard Attention Transducer using SED alignments as an curricular guide.
A lot of these things won't necessarily click, but I believe adding this new modularity layer would allow easier use of curricular learning and exploration scheduling that isn't easy to implement in other libraries. Expanding utility.
(This is down the roads thought. Post-beta.)
Something I've been thinking about with expansion of library: a decent amount of the work we've been using involves application of inductive biases and teacher-prompted training to model architecture. Currently we have:
One thing I would like to do with the next overhaul is modularize these beyond their respective models (like we're trying to do with #77 for teacher forcing) so that they can be 'dropped in' wherever. This would allow 'fun' combinations such as:
A lot of these things won't necessarily click, but I believe adding this new modularity layer would allow easier use of curricular learning and exploration scheduling that isn't easy to implement in other libraries. Expanding utility.
(This is down the roads thought. Post-beta.)