They did mention that the inference time to answer a question about the book was something like 22 seconds, so perhaps they are indeed still using self-attention.