Abstract: Language Models (LMs), in their most basic form, perform just like any other machine learning model - they produce interpolations and extrapolations based on their training distribution.