# On Multi-Armed Bandits and Normalization of Raw Counts for Feature Ablations

On Multi-Armed Bandits and Normalization of Raw Counts for Feature Ablations What up world. It's been a normal amount of time between blogs and I was in a mood—so I figured writing would be a cathartic and productive use of my time.

I've been delving into system design fundamentals for the past couple of hours, which is a long overdue hill for me to surmount. This blog post doesn't really have a concrete objective, so if it seems like a mixed bag of disparate ideas—honestly, that's what it is.

Multi-Armed Bandits and Thompson Sampling

A few weeks ago I was following the rabbit hole of reinforcement learning techniques because they're interesting and a little esoteric compared to the typical domain I focus on. Are math hipsters a thing? If not, I'm starting the bandwagon now (feel free to join).

Basically, multi-armed bandits are a reinforcement learning optimization technique. They let you pick an objective to optimize and update what's called a prior—which is like your expectation of outcomes prior to starting an experiment.

Think: prior to running an experiment.

These things are Bayesian, and they update your prior with observed data to optimize some kind of loss function, which is the actual learning part. My friend Mauro recommended a really cool book called Pattern Classification by Duda, which exposed me to them a few months back. I've wanted to try them for a while, and I thought of an idea—and sure enough, Google beat me to it like 20 years ago (shocker).

I was interested in optimizing click-through rates with them, so I did some digging and found a particular variant which relies on Thompson sampling that I thought could be cool to highlight.

Whenever you go on a dating app (I do not go on dating apps because I am happily spoken for), and set up a profile, this is the tech that powers those smart galleries which update your top photo based on interaction data. So if multiple people swipe on your photo of you next to your friend's boat—and the platform sets that misleading photo as your default—this is commonly the learning algorithm to blame.

How They Work

Bandits are agents—they are processes that explore and exploit to try to optimize a reward. They have arms or policies that they periodically test (the frequency to test depends on a probability distribution, i.e., Thompson sampling). Whichever arm yields the most reward is frequently shown the most. However, the cool thing about these guys is that they will still occasionally explore other options.

The reason this is so relevant in e-commerce is that behavior of what's popular or what maximizes the reward of clicks, purchases, or whatever action can always change in the future. It would be short-sighted to set a belief once and then hold that static forever. So these things can use weights to update the likelihood of sampling an exploratory policy versus exploiting the current popular ones.

Implementation

To help make this more clear, I'll write out some sample Python that you might use to initialize one.

For all intents and purposes, let's assume we're batching this for performance limitations (otherwise these are typically not batched).

alpha_prior, beta_prior = 1, 100

class MultiArmBandit:
    def __init__(self, arms: list[dict]):
        self.arms: list[dict] = arms
        # an arm looks like this:
        # arm = { 
        #   url: str = "", 
        #   alpha: float = alpha_prior, 
        #   beta: float = beta_prior, 
        #   confidence: int = 0  
        # }

    async def update_beliefs_from_observations(self) -> None:
        # Fetch observation data from external service
        events = await request.get(url='real-fake-page-views.com')

        arms_by_url = {arm.url: {
            'shown': 0,
            'clicked': 0,
            'alpha_prior': arm.alpha,
            'beta_prior': arm.beta,
            'confidence': arm.confidence,
        } for arm in self.arms}

        # Count events by arm
        for event in events:
            if event.type in arm_by_url[event.img_url]:
                arms_by_url[event.img_url][event.type] += 1

        # Update our arms based on beliefs
        # Posterior = the updated probability distribution of an arm's reward 
        # parameter after combining prior beliefs with observed data.
        # 
        # new_successes = clicks
        # trials = times shown
        # failures = trials - successes
        # alpha = prior_successes + new_successes
        # beta = prior_failures + observed_failures
        # confidence += 1 (higher = more data-driven)

        for url, arm in arms_by_url.items():
            new_successes = arm['clicked']
            trials = arm['shown']
            failures = trials - new_successes
            updated_alpha = arm['alpha_prior'] + new_successes
            updated_beta = arm['beta_prior'] + failures

            # Find the actual arm on instance
            actual_arm = next(
                (a for a in self.arms if a.url == url), None
            )
            if actual_arm is None:
                continue

            # Update the arm
            actual_arm.alpha = updated_alpha
            actual_arm.beta = updated_beta
            actual_arm.confidence += 1

    def thompson_sample_from_arms(self) -> str:
        import numpy as np

        samples = {}
        for arm in self.arms:
            arm_id = arm.url
            samples[arm_id] = np.random.beta(arm.alpha, arm.beta)

        # Pick arm with highest sample
        best = max(samples, key=samples.get)
        return best

The above code gets you out of the gates with a bandit initialized, assuming you have the ability to call real event data via some kind of analytics service. I'm not going to include logic for how you would use this—we can all use our ImAgInAtIoN on that part.

I do want to highlight that the purpose of this code is to illustrate how simple these are to construct and how effective they can be when you're faced with a known unknown. Here's a blog article I found helpful for making this less foreign to me:

Thompson Sampling for Bernoulli Bandits

Feature Ablations and Why Normalized Features Beat Raw Counts for Understanding Trends Recently I had the opportunity to do some research—I can't discuss specifics about what the results were and don't really want to go into too much detail about the objective—but to be vague and still somewhat coherent, I was trying to improve R² for a gradient-based tree regressor.

I set up a test which replicated our production pipeline and its features. I was trying to measure whether an aggregated feature (which in some ways contained information we already had in our pipeline) helped or hurt R². The outcome was not initially promising, but an interesting thing I learned along the way was this:

When you're trying to understand trends for phenomena across multiple environments that differ in scale, using normalization to your data first—to understand what your data means relative to the scale of its neighbors—can get you closer as a starting point than relying on raw counts.

I know that sounds vague, so let me illustrate with an example.

The Perfume Demand Problem

Say you're trying to understand how people buy perfume with respect to time in a given part of the US. This is a complicated problem. For one, perfume differs by olfactory segment. There are niche perfumes like Toskovat and then there are your more mainstream Angel Share or Marc Jacob's Daisy scents, which are purchased way more frequently.

It would be naive—and statistically incorrect—to assume that the monthly sales of a perfume like Anarchist A, which smells like "Dirty Dollars, Ink, Candle Wax" and is only sold in one store in SF (ZGO in the Castro), is representative of the velocity at which people are purchasing perfume as a whole in San Francisco.

A more meaningful question for understanding demand of perfume sales in a region is: What are the monthly sales for different perfume segments in that region?

The signal for all of them might differ. Seasonal perfumes might be ripping while oceanic scents might be in a downturn because summer's over and people are more tuned into smelling like Christmas trees and pumpkin spice than the ocean.

The Clustering Insight If you're reading this and thinking "hey, this sounds like a clustering problem first"—your instincts are right.

Finding out similar perfumes is something I'm working on doing here with graphs, sentence encoders, and TCVAEs. This oddly enough has led me to learn about mechanistic interpretability with LLMs and how polysemanticity makes generalization great but isn't great for representing distinct but similar notes in transformer embeddings.

This will probably end up being a future blog post on how I explored various VAE architectures to try and disentangle features from compressed embeddings and how those performed against manually created embeddings, which were monosemantic with every note getting its own vector.

Deriving Velocity and Normalization

For your time series sales data, you have a number of purchases for your cluster of perfumes that are similar. At this point you can try to derive the velocity at which they're being purchased.

Velocity = amount / time_interval

Another thing you can consider is using count sold versus total distributed (if this is somehow publicly available) to get a percentage—which, you guessed it, is normalizing raw counts.

This can help you predict whether there might be some consumer demand for a newer perfume that hasn't been reviewed or shared on social media yet.

The Search Visibility Factor

Note: search engine visibility is a really important feature to consider when understanding demand. People buy what they can find. So just because product A has similarities in its scents or branding to product B, if product A is shown way more, then the similarity between the two starts to lose its weight when you're considering it as a feature for estimating product B's value.

Suffice to say, consider picking a couple of popular sites like Fragrantica or Parfumo to get a baseline for how popular/visible a particular perfume is.

Also, a plug for Google's new Trends API—which they are releasing. This gives us access to understand time series data for how a query is trending on their search engine. Note: I'm not sure if Anthropic or OpenAI will also monetize trend data for their LLMs—that seems like a quick win though.

Normalization Techniques

So two normalization techniques you should consider when trying to estimate demand for a product are:

Z-score: The delta between your subset of data and the average standard deviation Percentages: Make isolated counts comprehendable across groups of counts that might vary in scale To back me up in this assertion, I'm going to drop some research from arXiv for you to check me on:

• An Empirical Analysis of Feature Engineering for Predictive Modeling
• Normalization in Proportional Feature Spaces

Hope you enjoyed my rants. And if you got this far reading, thanks! I hope you found some of this stuff useful