How are LLM’s increasing their context size? I guess you just increase input size if it’s for the self supervised GPT3 style training but for RLHF?
Are they creating datasets of books to input to the LLM and then making human labelers label the response? There might be a smart way that does not involve new datasets
Mosaic wrote about their new model here. https://www.mosaicml.com/blog/mpt-7b It was trained on 65k inputs and has decent performance working with 80k+ tokens.
