The Limitations of a Transformer

It's very expensive to make these models read really long text all at once. Another limitation is that models do a fixed amount of thinking or processing for every word they produce. We haven't figured out how to cleanly reliably separate kind of the easy cases where you can think quickly, what you might hear called, system one reasoning in cognitive science.