Extracting representations from the LM

[jonny-d]

Hello,

Is it possible to extract features from the language model using the current version of the code?
For example, given a string such as "the cow goes moo", can we extract a vector for use in downstream NLP tasks?

Thanks

[WuTheFWasThat]

There's no single obvious way to encode a sentence, but this is a good area for research! Lots of things I could imagine trying (some way of combining the embeddings per position, finetuning with a new head, etc). I'm not too familiar with existing literature here so others should chime in

[shamanez]

Any luck with this? Can we extract a representation for the sentence "the cow goes moo