Kevin Hillstrom: MineThatData

The Report Lies To You

Non-catalog readers, take the day off.

Once again, I share these images to show you just how wonked-up AI actually is. Spelling errors. Missing legs. If it's this wonked-up with a simple cartoon, imagine what else the gurus are trusting that in reality is completely borked?

But I digress.

Last week a member of the #printisback community on LinkedIn decided that #brands need to go back to 1996, as if J. Peterman were about to publish the Urban Sombrero on the cover.

The root of the lie being told to you is called "Matchback Reporting". Matchbacks ignore incrementality. Those who believe in Matchback Reporting ignore reality.

Time for a parable. Remember when I told you a member of the retargeting community took me to lunch, and for the cost of a Caesar Salad wanted me to convince the Management Team at a large retail brand that because 95% of ecommerce customers saw at least one of his ads he deserved credit for 95% of ecommerce orders (over $300,000,000 in annual sales) and he wanted a percentage of each transaction. If he presented those facts to you, a smart cataloger, you'd say "hey, Goober, get out you Lemonhead." Then you'd go back to your cubicle, pull out your Matchback Report, and perform the EXACT SAME ANALYSIS AS GOOBER DID and treat it as Gospel. You'd think that Goober was an idiot, you'd view you as a savvy marketer. And yet? You're the same person, doing the same thing, sans Caesar Salad.

Let's assume that your catalog is active across a four-week period. Let's assume you were going to mail your entire twelve-month customer file. You randomly select 50,000 customers to be in the mailed segment, you randomly select 50,000 customers to be in your no-mail / holdout segment.

You then sum all demand across all marketing channels for the four weeks when the catalog is active. Compute an average (i.e. divide each sum by the 50,000 customers in the segment). You'll produce a table that looks like this:

This is the way test results generally look. If you don't mail a catalog, your call center is quiet as 78 year olds cannot call you if they don't have a catalog in their hand. Website / Direct Load is where most demand happens, most of the demand will happen if you don't mail the catalog. Email Marketing is usually not impacted by catalogs, though your mileage will vary. Search is a channel that is clearly impacted by catalogs - catalogs cause customers to search for competing products (both a strength and a weakness of catalog marketing because your dumb catalog drives your smart customers to the competition). Social is almost never impacted by catalogs - completely different audiences.

The magic in the table happens in the Total Demand column.

Your matchback reporting takes full credit for the $5.65 of total demand. It ignores what would have happened if the catalog were never mailed. If the catalog were never mailed, the $5.65 total becomes $4.25 ... not $0, but $4.25.

Here's what the p&l might look like for the $5.65 figure.

Everything looks good here ... you appear to generate $1.54 profit per catalog/book ... in modern parlance you generate a ROAS of 7.53. It's the 7.53 figure that the #printisback community on LinkedIn like to refer to ... it's a much higher number than that 3.88 figure you get for paid search or 2.97 for paid social or ... wait ... they never quote email marketing because email marketing has the best ROAS, period.

Remember - your holdout group did $4.25, not zero. Take the control / holdout group average ($4.25), divide it by the mailed group average ($5.65), and you get 75%.

This means that 75% of what is outlined on the matchback report is a lie. A fabrication. It would have happened had the catalog not been mailed ... and you know this is true because in the table YOU DIDN'T MAIL THE CUSTOMERS IN THE HOLDOUT GROUP!!!!

We cannot run the p&L on the $5.65 that is likely reported in your matchback report. We have to run it on the incremental total ... $1.40 ... which is (1 - 75% = 25%) of the $5.65 total.

The p&l changes, friends.

This is where things get really dicey. The incremental outcome (a loss of $0.18 per catalog) is REALITY. The matchback-reported totals column of $1.54 profit is FANTASY.

On LinkedIn, the #printisback community communicates FANTASY results to you, misrepresenting the outcome as reality.

It's pretty obvious why they'd do this.

$5.65 per book / $1.54 profit = Best ROI, which means you should hire them.
$1.40 per book / ($0.18 profit) = It might be time to shut down your catalog division.

When I communicate (via analysis of thousands ... seriously ... of catalog mail/holdout tests across 36 years) what a smart catalog brand should be doing, I'm generally derided by the #printisback community, and for good reason. I want you to do what is most profitable for your business, they want you to do what is most profitable for their business. Do you see the distinction there?

So, yes, the #printisback community is probably right ... I'm not one of them as they tell my clients.

But I'm on your side. I want you to be as profitable as possible. Matchback reports lie to you, and somebody has to communicate that to you.

Feeling Special

I gave a presentation in 2016, in front of about 1,500 people. I spent nearly an hour explaining to the audience how running an ecommerce brand would become comparable to running a sports franchise.

The audience was not impressed.

I do recall a pair of Associate Athletic Directors working for FCS Colleges reaching out to me to suggest I "had it right" - they suggested their world was heading in this direction as well. My industry didn't agree, these people who weren't in the industry believed in the thesis.

Here we are, in 2026.

As you can see, AI has a way to go. That's a catastrophic effort at creating a cartoon for me. And you need to see the cartoon to understand AI limitations. How will you know when AI completely butchers your marketing efforts? It's going to happen, and it's going to be spectacular.

Last year I stood in a luxury store with an Executive. The Executive told me to watch the customer. The customer was spending somewhere north of $5,000 ... and she had a glow that reached from Phoenix to El Paso. It was the kind of glow an Eagles fan might feel after beating the Cowboys 34-28 on a last-second touchdown pass.

The customer felt special.

The theoretical Eagles fan would feel special.

It isn't hard to see the future of ecommerce bifurcate.

AI-Agents shopping on your behalf at AI-Marketplaces. That's cold and boring and sterile.
Brands who make customers feel special.

To get back to the sports analogy, you're probably going to hire a General Manager at some point in the future.

AI-Marketplace Director reports to this person.
AI-Agent Director reports to this person.
Ecommerce Director reports to this person.
Creative Director reports to this person.
Website Operations reports to this person.
Digital Marketing Team reports to this person.
Merchandise/Marketing Communicator (Director) reports to this person.
Analytics Team reports to this person.

The General Manager (GM) brings everything together. She's not unlike Billy Beane in Moneyball. She has three major job functions.

Partner with merchandising/inventory leadership to put the best products in the best situations to have the best outcomes ... similar to what a baseball GM does with A / AA / AAA / Major League players. There's a development plan for each player. There needs to be a development plan with winning/new merchandise that is marketing-driven.
Lead the company into the future with AI, setting aside all the hype and grifting, focusing on what matters to customers.
Make customers feel special. This doesn't mean offering somebody 40% off. This doesn't mean triple-loyalty-points. This likely means creating digital and offline events that fill an emotional need with the customer. This isn't fundamentally different than six Saturday home game afternoons during the College Football season.

There are going to be new companies that create the AI-Marketplaces of the future. We have no idea who those companies are (they likely don't exist yet or they have 11 employees), we don't know what AI-Marketplaces will look like. We don't. Anybody who tells you they know is a Thought Leader. We will need a General Manager position to cut though the grifting that will happen or is currently happening.

The GM needs to also make customers feel special.

Spend An Hour With Me. And Daniel. And Aaron. On Monday

Join Daniel/Aaron from Orita.ai and I on Monday at 4:30pm EDT / 1:30pm PDT as we talk about ecommerce and bridging the gap between Executives and those of us with facts that need to be acted upon.

Click Here, now!!

In 1992 Your Promotion From Analyst To Manager Was Published in DMNews

By 1994 my promotion wasn't published anymore. By 2026 you self-published your promotion on LinkedIn (and earned 77 "likes").

But 1992?

In 1992 you'd submit org structure changes to DMNews so that you could communicate to the entire Catalog Industry just how sophisticated your intentions were.

"LL Bean announced today the promotion of Shannon Ellison to Director of Circulation. She brings with her nearly eight years of experience. Ellison mentioned that she's '... looking forward to partnering with our printer to bring perfect binding to the Christmas catalog'. At press time, LL Bean has not decided whether the Christmas catalog will be 256 pages or 260 pages."

These announcements meant something. There was actual gossip ... "do you think LL Bean could go to 264 pages, I mean, Ellison likes a meaty assortment from what I've heard." I recall our restructure in 1992 to a "Customer Planning and Development" framework (or maybe we left that framework, I don't know). We announced a new Director and new Managers. My goodness, the stimulating conversations as LL Bean and Lands' End held an "exchange" meeting (to exchange names with each other) in room 318 at the Marriott during the Catalog Conference, with 18 people (including 6 actual employees from LL Bean and Lands' End paired with 12 kind vendor-supporting staffers) sitting around two queen beds negotiating the trade of 1.6 million names at $0.005 each.

Twenty years later? All of it ... gone. All of it. Replaced by Google.

Imagine everything that exists today ... gone in 2046. It's going to happen.

Viable Assortment

Here's one of the optional analytics that I might add to a Top 12 project ... I call it the "Viable Assortment".

Viable Assortment = The Number of Items/Styles You Sell That Generate Enough Volume To Be In The Top 90% of Your Sales Assortment (Annually).

The items in the bottom ten percent of your assortment are just rubble ... stuff that is being cleared out, stuff you sold three years ago and 17 customers still love it.

The items in the top 90% of your assortment are your "Viable Assortment". It's what customers care about.

Here's a company that is failing. Here is their "Viable Assortment".

End of 2025: 1,206 Items, Price = $18.20, Average Sales/Item = $15,520.
End of 2024: 1,158 Items, Price = $15.21, Average Sales/Item = $19,892.
End of 2023: 1,335 Items, Price = $15.72, Average Sales/Item = $18,471.
End of 2022: 1,573 Items, Price = $16.34, Average Sales/Item = $16,273.

Weeeeeeee!

Among the Viable Assortment, this brand jacked up prices (or introduced expensive items ... hint hint), driving down Sales/Item in the process. And compared to three years ago, we see that the Viable Assortment is 20% SMALLER than it is today. Bad. Not smart. That's 370 fewer items generating at least $15,520 less ... that's $5.7 million that evaporated due to a smaller Viable Assortment.

Do you measure your Viable Assortment?

It can be part of the "bonus analysis" that is included in the Top 12 Project ... contact me now (kevinh@minethatdata.com) to take advantage of the introductory offer for prior clients and blog subscribers ... offer ends August 15.