Item Details

Examining the Performance of Different Contextual Representations in a Canonical Language Model

Jacques, Brandon
Thesis/Dissertation; Online
Jacques, Brandon
Sederberg, Per
Driven by advances in computer engineering, model architectures, and training methods, the field of Natural Language Processing (NLP) has reached new heights of performance. Currently utilizing short-term buffers to represent the context in which a word is experienced, these models have lagged behind recent developments in the understanding of human memory. Finite representations of context ignore the fact that the temporal scale in which words are predictive of each other can go up to hundreds of words apart(H. W. Lin & Tegmark, 2017). In this paper, we leverage recent developments in the understanding of memory to augment the performance of a canonical NLP model with a compressed representation of context that contains many time-scales of information. We show that the Timing from Inverse Laplace Transform (TILT) representation, a neurally plausible way of compressing history utilizing leaky integrators, can function as a drop-in replacement for a buffer representation in a canonical language model to increase performance without adding computational complexity or increasing the size of the overall model
University of Virginia, Psychology - Graduate School of Arts and Sciences, MA (Master of Arts), 2019
Published Date
MA (Master of Arts)
Libra ETD Repository
Related Resources
Artificial Neural Networks Statistical Language Modeling Compressed Memory Neurally Plausible Representation
Logo for Creative Commons Attribution LicenseCreative Commons Attribution License


Read Online