### Transformers and Recommendations

Hey world, it's been a minute - i figured i'd take some time to reflect on what i've learned over the past few months as their has been a lot of change in the scope of my work on a day to day basis.

Recently, I read an article from patrick nicholas (who has a great substack on geometric deep learning) on the math behind consciousness - and whether it's posssible to model the neural processes that allow humans to be conscious and learn. In the article, one of the main concepts that he explores is how compression might be akin to learning. He prefaces his exploration of this idea, with a hefty - not backed by neuroscience yet caveat, so i feel it necessary to do the same. However, as I learn more about deep neural networks - transformers - and two tower rec systems this idea feels increasingly more relevant - at least in the current landscape of machine learning.

I've been working on creating my own deep learning library, i'm dubbing clem - which is essentially a far worse performing, less robust and redundant version of pytorch (or karpathy's micrograd). I'm doing this to get a richer understanding of modern ml systems but also to be able to understand the underlying operations that occur inside a transformer which are usually obfuscated by convienent abstraction. Doing so has helped me turn the corner conceptually and technically on transformers. I've started to get how multi head attention works mathematically, how transformers project positional encodings over their vocabulary of tensors to create a contextual understanding of sentences in addition to having a separate understanding of each word. Transformers combine knowledge (via tensors) in an additive way (when projecting positional encodings onto stacked input sequences inside of their input layers), and in a product way - with dot product operations for deriving meaning from Query, Key, Value tensors that allow them to understand neighboring words in a sentence via attention. From a high level, the way GPT-2 (specifically) understands language by combining different tensors is a surprisingly simple idea; albeit still difficult to implement.

This simple realization about how transformers understand language is also found in recommendation systems, like two tower neural networks which compress tensors of item features with tensors of user behavior features in order to learn. After training, the forward pass of a two tower system uses a dot product between a given vector representing a user and the learned embeddings of the model to understand what items they might be inclined to purchase.

At the core of both systems in two entirely different domains of deep learning lies a simple dot product operation. Multiply column i by row j over the whole tensor (or matrix) add them up, then the resulting matrix (or tensor) contains an understanding of both.

Also, on the subject of recommenders - i'm happy to say that i'll be attending this years rec sys conference in Minneapolis. I'm going to be participating in some cool work shops on travel recommenders and modeling sequential recommendations in order to accomodate for how people's taste change over time. I think a lot of the conference seems to delve into how industry and academia are using transformers for recsys so i'm hoping i can walk away with a richer understanding of how to use the two together more effectively.