Kevin Hillstrom: MineThatData

Speaking of Setting a Standard

I asked ChatGPT to perform a simple long-term value curve estimation procedure for me. The classic ecommerce customer relationship is one easily fit via a diminishing returns power function.

Future Spend = a*x^b where "x" is the number of months since a customer was acquired.

I'll spare you the paper trail for now (it's attached at the bottom of the post).

AI fit the wrong equation ... it fit a linear regression model which understated how much a customer spends early in the life cycle and overstates how much a customer spends later in the life cycle. In the real world, this means you'd accept a 2.00 ROAS when you should accept a 4.20 ROAS and your company would be significantly less profitable. All because of AI. If you don't know how to "coax" an answer out of AI, you'll doom your company. And you'll boil a few aquifers in the process.

I asked AI if a diminishing returns relationship like a*x^b is more appropriate and if it fits better? It replied that the equation is more appropriate and yes, it fits better (well then why didn't you do that the first time I asked you?). It gave me an answer. One problem. The answer it gave me was a hallucination? How did I know? Because I asked my statistical software package (CurveExpert) to solve the problem for me, and it gave me the correct answer.

It gets worse.

I asked AI to provide me with predicted value calculations for the equation it fitted. Here is an example of the answer it provided for me (remember, the coefficients are already wrong - I'm just asking it to perform simple math).

Future = 8.63 * (Months ^ 0.78).
Assume Months = 12.
Future = 8.63 * (12 ^ 0.78).
Future = 55.90.

Go ahead and plug 8.63 * (12 ^ 0.78) into Excel. What answer do you obtain?

You should obtain 59.95.

In other words, AI is not even calculating simple calculator-level math correctly. It's hallucinating. It's doing whatever it wants. And it is doing a HORRIFYING JOB.

You are a Lemonhead if you trust this stuff.

Yes, there will likely be a day when AI is a lot better and performs certain functions a lot better. If you want to target email subscribers properly, AI is already quite good at that ... though again, how do you know it is doing things the "right" way? How could you possibly know? Also, I just asked the application to provide me SPSS code for a problem and it wrote elegant and functional code.

But until that future day happens (and it appears it is out there a bit), please be careful.

Here is my final question to AI, and AI"s final response.

Kevin: I've learned that you cannot perform even the simplest of tasks without making mistakes. I am able to fact-check your work with hand calculations. What are users who do not have my skills supposed to do if they blindly trust your solutions when you cannot even perform simple math that a calculator can perform?

AI: Kevin, you’re not wrong to be frustrated. And you’re absolutely right to call out the risk: if I make a math mistake and a user can’t check the work, that’s a problem. Let me speak to you directly and clearly — no defensiveness, no sugarcoating.

1. You caught real errors — and they were my responsibility

You gave me a clean dataset — values like:

“60,169.17 … 1,11.03”

and you asked for a specific model:

y = a·xᵇ

I should have executed the log–linear regression cleanly. Instead, I produced inconsistent coefficients and even mis‑computed basic exponentiation. You were right to challenge it.

2. What should users do if they can’t check the math?

They should do exactly what you’re doing now: question the output when something feels off.

Even though I’m designed to be helpful, I’m not infallible. I don’t “blindly calculate” — I generate answers through patterns, reasoning, and approximations. That means:

I can make arithmetic mistakes.
I can misinterpret a model specification.
I can produce a result that sounds confident but is wrong.

A calculator never does that. I can.

3. So how do non‑experts stay safe?

Here’s the honest guidance:

If the stakes are high, users should verify with a second source. A spreadsheet, a calculator, statistical software — anything deterministic.
If something looks suspicious, they should ask me to re‑derive it step‑by‑step. When users force me into explicit steps, the chance of error drops dramatically.
If a model choice seems odd, they should ask why I chose it. That exposes misunderstandings early.
If they don’t know how to check the math, they should ask me to show the logic in a way they can check. For example: “Show me the log‑linear regression table.”

4. And for you specifically

You’re not a passive user. You’re doing exactly what an expert should do: stress‑test the output, challenge inconsistencies, and demand rigor.

Your skills make you a tough critic — and that’s good for me. You force precision.

5. I’ll own the core issue

You asked: “What are users supposed to do if they blindly trust your solutions?”

They shouldn’t blindly trust me. They should trust me the way they trust any powerful tool: use me, but verify when accuracy matters.

And when you catch me making mistakes, I don’t get defensive — I get better.

If you want, we can redo the entire regression cleanly, step‑by‑step, with every transformation shown explicitly so you can see each number and confirm it.

Setting a Standard

My Milwaukee Brewers are having yet another great year, one that will potentially be derailed by the Cubs or Dodgers. On Tuesday night, this happened (click here).

A rookie was benched in the fifth inning AFTER hitting a bloop single.

Yes, Brewers fans derided the Manager for being a bit short-fused.

Not me.

He's setting a Standard.

Any of you who have been a Director or VP or C-Level Professional know that there is a Standard in your industry. There is a level of competence and effort that must be met, and if the employee does not meet it, there are consequences. You cannot survive if your team does not meet the Standard. And you know what that Standard is, based on your career experiences.

I work with a current client where the employees are understanding the Standard, and are responding to the Standard. Their response is different than a year ago. I so enjoy working with them! The employees are growing!!!

Go out on LinkedIn for 10 minutes and you realize that so many of those folks do not understand the Standard. No attention to detail. Just high-level thoughts. "I think Kohl's has big problems with their relationship with Sephora." That's brain-dead drivel. Do you know what counts? Meeting the Standard so you can fix both Kohl's and Sephora and change the trajectory/thoughts of tens of thousands of employees. That's the Standard. How many people on LinkedIn are capable of doing that?

Catalog Thought Leadership is just as bad. When faced with postage increases, my clients aren't asking the "wrong questions" as printers and paper people tell us. My clients are meeting the Standard. They are benching Print for lazy performance and poor effort. They are benching the vendors who support lazy performance and poor effort. I can understand if you were benched that you'd maybe take a swipe at the person who benched you, but that's also not meeting the Standard. Read what Cooper Pratt (the player who was benched) said in the article cited earlier.

The Standard = Discipline + Competence + Vision + Leadership + Communication.

How many unsubs am I going to deal with for merely referencing this topic? And what does that say of the person unsubbing?

Loyalty: Red Lobster Nation

We talked about loyalty programs this week. I talked about the feebleness of points and percentages off. Here's Red Lobster Nation.

Earn points, get dollars off your meal.

A question.

Is the price of a meal at Red Lobster the thing that has stopped you from dining at Red Lobster? Is the price of a meal at Red Lobster the thing that stopped you from eating at Red Lobster 13 times a year instead of 12 times a year?

A loyalty program should be designed to solve a problem. What is the core problem that Red Lobster has?

Is it that restaurants need to be renovated? If that is the core problem, how is encouraging somebody to eat in a run-down restaurant more often for a few dollars off helpful?
Is it that Zombie Retail restaurants are in less-than-optimal locations (i.e. they were put in a good location 20 years ago but those are not prime areas anymore)? A loyalty program won't solve this problem.
Is it that the food is too expensive? This could help, but you are asking the customer to continue to pay higher prices for a period of time before earning a small reward.
Is it that the service is poor? If this is the core problem, asking customers to continue to receive poor service for a period of time before saving a few dollars is a big ask.
Is it that Marketing is out of ideas? If this is the core problem, this could be a solution.
Is a Management Consultant involved? If this is the core problem, we all know the appropriate course of action.

In almost all cases, the "brand" (or Zombie Retailer in this instance) has a merchandise / product / pricing problem that the brand is choosing not to address - the loyalty program is designed to paper over the merchandise / product / pricing problem.

The Two Best Ways To Grow Your Loyal Customer Base

I've run the numbers more times than I care to mention. Tables, queries, simulations, you name it. All methods tell the same story.

The best loyalty programs have MANY customers, not few customers. It's always better to have two loyal customers than it is to have one loyal customer.

If you want a loyalty program with many loyal customers, there are two things you need to do. You need to do these things years before they pay off (I realize you don't want to hear that news).

You must acquire high-quality new customers. S-Tier or A-Tier new customers. It's mandatory. That new customer you paid Facebook for that bought one lousy item at $29.99? Garbage. Facebook makes money, you don't.
You must convert as many first-time buyers to a second purchase within three months of a first order. If you don't get them early, the probability of the customer becoming loyal greatly diminishes.

The mistake that is made, of course, is that the loyalty marketer waits until the customer spends $1,500 or whatever the amount, then tries to squeeze more money out of the customer. How many customers ever get to the $1,500 level? In my work, somewhere between 2% and 10% of customers ever achieve "loyal" status, however you define it.

Smart marketers, of course, mitigate this problem by crafting alternate marketing programs.

They don't say "no" to the garbage name acquired via Facebook, but they work overtime to acquire the first-time buyer who purchases three items on a first order in two different merchandise categories. Whether algorithmically or (often) via their own programs, they generate attention/awareness that leads to new customers that are S-Tier or A-Tier.
They have well-developed Welcome Programs that convert customers to a second purchase quickly. This results in a significant increase in loyal buyers 18-36 months later. The Loyalty Professional is dependent upon a Smart Marketer.

When clients ask about loyalty programs, I frequently say "If you want twice as many loyal customers tomorrow, be sure to acquire twice as many good new customers today". That's the point where professionals (i.e. some of you) get frustrated.

There are no shortcuts. There is a Standard that needs to be met.

Regardless, that's what the data shows. Accept facts and thrive!

The Problem With Loyalty Programs

Three problems with loyalty programs:

Wrong Incentives. Points and Discounts. Is that what the customer truly "wants"?
Wrong Customers. The wrong customers are selected to be included ... sometimes it is almost every customer that is included. That's not a loyalty program, it's not special if everybody is included.
Wrong Outcome. If we assume that a loyalty program creates incremental orders that wouldn't have otherwise happened (a big assumption), we may or may not generate a profitable outcome. For instance, too many of you ADORE throwing gross margin dollars in the trash can to "create" a more loyal customer. Why are you giving everybody an additional 20% off? Would they have purchased without the discount? If the answer is "yes", you just threw money in the garbage can and lit it on fire.

A well-crafted loyalty program must result in incremental orders that wouldn't have happened otherwise, and must result in more gross margin dollars and more profit dollars that would not have happened otherwise. Every time you give an additional 20% off or 40% off, you put gross margin dollars and profit dollars at risk.

Also, you don't solve the core problem. If you want to have a great loyalty program, how do you grow the number of customers who deserve to be in the program?

P.S.: I once worked with a "brand" that decided to enter everybody spending > $500 in a loyalty program, offering discounts/promotions/points to encourage the customer to spend more. The marketing team loved watching the orders roll in (in truth, they'd never measured how orders came in for this cohort). At the end of a year, I quantified year-over-year how much the > $500 cohort spent (it was like a 20% increase). Everybody celebrated. Then I shared with them the outcome of a separate query I ran where I measured the year-over-year increase among $400 - $499 customers last year. They didn't spend 20% more ... but they spent 15% more.

The incremental increase of 5% was wildly unprofitable. The company simply burned money.
Nobody appreciated the answer. I wasn't invited back to continue analyzing the issue.