Whenever I buy ice cream at the supermarket, I’m overwhelmed with the number of choices. Which one is the best? Does how hipster it looks have any impact on the quality? Ariel Liu and I finally decided to settle it once and for all with an 18-way grudge match by inviting 20 of our friends over.
Methods
To keep things comparable as possible, we decided to get plain chocolate ice cream from every brand. To anonymize the packaging, we scooped the ice cream into solo cups and passed them around. Each ice cream had a pokémon codename. Tasters stack ranked the ice creams rather than assigning an absolute score to avoid the problem of all the good ones having to similar a rating. There was no explicit rating criteria except “Which do you like best?” We recommended the tasters consider things like flavor and texture to keep track, but it wasn’t required or tracked. We allowed tasters to request a re-taste to refine the ranking as necessary. The palate cleanser was water. The ice creams were mostly tasted in the same order for every taster, though people who came in late got the first few last.
Caveats
We couldn’t control for temperature well due to our friends bringing the ice cream in various states of melting.
Ordering might matter. By the 18th ice cream, I imagine tasters were pretty fatigued. That said, some ice creams at the end did receive high ratings, including the winner.
Who makes the best chocolate ice cream doesn’t necessarily translate to who makes the best other plain flavors, or who has the highest quality mix-ins (Ben and Jerry’s?).
Results from best to worst
We also had Salt and Straw and Tillamook, but had to disqualify them since they weren’t plain chocolate. You might also disqualify Jeni’s, which had coffee flavoring added, and Coconut Bliss, which is non-dairy. We’re missing Bi-Rite, Edy’s, Ben and Jerry’s, and Breyers, plus a bunch of regional favorites.
See the spreadsheet for tasting order, prices, and taste-affecting nutrition facts.
Discussion
Most people expected a local ice cream shop to win instead of Trader Joe’s (aka Double Rainbow). It’s possible that tasting order influenced it, but the previous ice cream was Coconut Bliss, which got a pretty good rating. It even had the lowest variance of any rating, so everyone agreed it was great. I’m guessing that’s because it had a very obvious chocolate flavor that wasn’t at all artificial. It’s hard to vote highly for a chocolate ice cream with a weak flavor, and most of the other store brands suffered from this. If we were voting for vanilla it might have come out differently. It almost seems like Trader Joe’s flavor is optimized to win tasting competitions, even if you wouldn’t necessarily reach for it if it were in the freezer next to something more primo. It was tasted 15th. Serious Eats also rated it #2 behind Friendly’s, a brand we didn’t taste.
Ratings were pretty consistent. The mean was nearly linear, indicating there was consensus on the overall ranking. The average standard deviation was 4, indicating most people were only different in their rankings by ±4 positions. The most divisive brands were Safeway and Straus; the least divisive were Trader Joe’s and Mitchell’s. Except for Enlightened and Talenti, every brand was in someone’s top 3. Except for Trader Joe’s, every brand was in someone’s bottom 4.
Another big surprise was McConnell’s coming dead last. I’d think a $9 pint would do better.
Some of my friends have told me they think diet ice cream like Halo Top tastes just as good as normal ice cream. I think it tastes much worse, so I was glad to see Enlightened near the bottom of the rankings. I assume Halo Top would fare the same. If you want to eat diet ice cream, please do, but we now have consensus agreement that it doesn’t taste as good. Someone please solve this.
Anecdotally, people hated Salt and Straw and liked Tillamook. I think that if Tillamook had a plain chocolate flavor they would have done well, but probably not beat Trader Joe’s.
Takeaways
- Shop at Trader Joe’s more
- Tillamook is as good as I thought; should be my go-to supermarket brand
- Coconut bliss is edible
- SF local shops (Mitchell’s, Xanath, Smitten) are good, but not necessarily better than store-bought. Go for the experience and the variety.
In the end, each person’s own tasting notes are the most valuable signal to them, since they’re the ones who are going to eat the ice cream next. But it was great to see that there is some consensus on what makes a good ice cream. I’ll know for next time whose recommendations to trust.
