A/B Testing Online Review Incentives Via Email Campaigns

Garrett SussmanEmail Marketing, Reputation Management, Review Marketing0 Comments

AB-Testing-Online-Review-Incentives

In the past we’ve discussed the risks and rewards of incentivizing online reviews. While Yelp is notoriously strict and believes (wrongly) that even asking for a review biases it, other review site publishers are more, um, rational. Sites that have a predominantly professional audience recognize that their reviewers’ time is valuable. Providing a small reward for their time can actually motivate more unbiased reviewers to participate. But it has to be done the right way.

Last week, we collaborated with the awesome B2B software review site G2 Crowd on an email campaign to get our latest crop of Grade.us customers participating in writing reviews. Because honestly, if an online review marketing and management platform lacks online reviews, then how can you trust their value proposition? 🙂

G2 Crowd actually encourages and facilitates providing reviewers a small reward to review products like ours. With review incentives being a hot topic–and one that’s misunderstood or over-simplified by both proponents and detractors–we saw an opportunity to A/B test the effectiveness of incentives on online review acquisition.

We had a few questions that we wanted to answer:

  • Does mentioning an incentive in the subject line of the email increase open rate?
  • Would the inclusion of an incentive in the email increase the click through rate?
  • Would we earn more reviews via incentivized emails versus non-incentivized emails?
  • How would the star ratings compare from incentivized reviews versus non-incentivized reviews?

Tick Tick Tick Boom!


We frequently talk about how it’s important to have a stream of new online reviews for our businesses, yet not all business models make this strategy easy. Velocity of online reviews, does play a role as a ranking factor for Google.

If our business model consists of one time transactions, based on products or projects, we can trigger email drip campaigns or SMS text campaigns for getting quality reviews at the time of sale or at the completion of the project. But if our services come on retainer, or in the case of Grade.us, an ongoing subscription model, finding the right time to ask for the review can be a bit more tricky.

Using email campaigns is an effective marketing tactic, one of many recommended by WeeklyM, a marketing strategy website. While it’s not as ideal as a steady stream of reviews, we’ve found value in sending out email blasts to a portion of our customers at a time.

Our Past Review Acquistion Email Campaign


Prior to our G2 Crowd experiment, we sent out our own email blast to a selection of new customers, to bolster our Capterra.com listing.

At the bottom of our Product Update newsletter, we dedicated the last section of the email as a dedicated call-to-action for new Capterra reviews. Here’s the copy we used:


Finally, << First Name >>, I need some help…

Here we go… dogfood time: You know how much customer reviews mean to us, right? Reviews are our business, after all. But for me, the value goes way beyond that.

I literally choke up when I read the kind things our customers say about Grade.us. And the critical things? They keep me up every night.

We’ve had some growing pains the last couple months, and as a result, I watched exactly zero March Madness games 🙁

But now one of our favorite B2B review sites, Capterra, is running its “Reviews Madness” promotion. And I can tell you that it would probably move me to tears if you’d write a review of Grade.us during this competition. It can be super quick–Capterra is easy. No signup. No fuss.

Seriously, I’d be forever in your debt–as would all those other professionals looking for a good review management solution on Capterra!


In the copy, our founder Jon Hall, appealed to our customer’s emotions. We’re not a massive organization, so it was essential for us to connect with our loyal customers in a personal way. Jon’s genuine words tugged at the heartstrings of our valued customers, and the results were beautiful.

Here are the results:

  • Our campaign went out to 223 customers.
  • Delivered successfully to 221 customers.
  • 149 people (67.4%) opened the email.
  • 35 people (15.8%) clicked a link from the email.
  • 13 New Reviews in 5 days.

Our 67.4% open rate is highly favorable compared to industry benchmarks provided by MailChimp. Industry benchmarks can be extremely helpful in comparing the success of our campaigns, but there’s just as much value by comparing results internally to past campaigns.

Incentivized vs. Non-Incentivized Email Campaign Experiment


With the same ultimate goal as our email campaign to earn Capterra Reviews, our email campaign experiment with G2 Crowd highlights both incentivized versus non-incentivized email campaigns, but also the difference in email campaigns coming from our company versus a third party.

Incentivized reviews are still polarizing, but most often it’s taken in the wrong context. Isn’t there a difference between incentivizing a positive review versus using incentives to simply earn a review regardless of its sentiment?

In the context of this experiment, we felt it was valuable to reach out to G2 Crowd to get a better understanding of this critical difference:

“G2 Crowd offers incentives because the conversion rates prove they work. Our internal research shows no significant difference in review sentiment based on whether an incentive was offered, across our site. We do not see a significant difference between the average net promoter score (NPS) reported by reviewers who were offered an incentive versus the average NPS reported by those who were not offered an incentive.

We make very clear that any incentive offered is a thank-you for taking the time to share their feedback, not payment for their time. It is also emphasized that the incentive is not dependent upon the sentiment of the review and that we are seeking only authentic, unbiased feedback. Additionally, G2 Crowd is an independent third-party requesting feedback; we are not motivated by the outcome of the review, only its authenticity.”- Molly O’Hare, director of content & community, G2 Crowd

For a business that is built on the authenticity of online reviews, we can imagine that G2 Crowd takes the side-effects of incentivizing online reviews seriously. They’ve done the research to back it up.

The email campaigns were sent out on July 27th. G2 Crowd sent out 161 incentivized emails and 161 non-incentivized on our behalf.

G2 Crowd kept the email copy simple, and we went in with a few adjustments so that the message stayed ‘on-brand.’

For the incentivized customers, we offered a $25 Amazon gift card through G2 Crowd.


Subject Line: Review Grade.us, Get An Amazon Gift Card

Hi *First Name*!

I’m reaching out in the hopes you could contribute a review of Grade.us on G2 Crowd. Grade.us asked us to contact you and its other customers to obtain unbiased, authentic reviews of its product.

To say thanks for taking the time to share your feedback, we’re offering a $25 Amazon.com gift card* to the next 20 Grade.us users who complete a review. The gift card is from G2 Crowd and your eligibility to receive it is in no way conditioned on the content of your review.

You can add your review of Grade.us here.

If you have any questions about G2 Crowd or the review process, don’t hesitate to ask.

Thanks for your help,

Molly


And here’s the email we sent out without an incentive:

Subject Line: ‘Looking for your feedback on Grade.us’

Hi *First Name*!

I’m reaching out in the hopes you could contribute a review of Grade.us on G2 Crowd. Grade.us asked us to contact you and its other customers to obtain unbiased, authentic reviews of its product.

You can add your review of Grade.us here.

If you have any questions about G2 Crowd or the review process, don’t hesitate to ask.

Thanks for your help,

Molly


The Results of the Campaign

The email campaign generated a total of 17 reviews, which exceeded our expectations, knowing how difficult it can be to actually get the reviews.

Non-Incentivized campaign

Reviews generated: 6

Incentivized Campaign

Reviews generated: 11

We found some interesting results from this experiment.

Open Rate

Emails originating from Grade.us were opened more than the campaigns from G2 Crowd. Our Capterra campaign saw an open rate of 67.4%, whereas the G2 Crowd non-incentivized and incentivized emails had an open rate of 41.7% and 54.9% respectively.

Insight:

It makes sense that a personal email from the brand would generate more opens than a third party email. Additionally, including the “Get an Amazon Gift Card” in the subject line had a positive impact on opens.

Click Through Rate

Despite being a platform feature update, our Capterra email campaign produced a welcomed 15.8% click through rate. The G2 Crowd non-incentivized and incentivized emails produced a click through rate of 8.6% and 20.3% respectively.

Insight:

Incentives seemed to have the strongest impact on click through rate. While the Grade.us email had the highest open rate, the incentivized emails produced the highest click through rate. It appears that the Amazon Gift card offer helped get more customers over to our G2 Crowd listing.

Final Review Acquisition Results

Here’s where it gets very interesting:

  • Capterra Campaign: 35 unique clicks generated 13 reviews, a 37% rate.
  • Incentivized G2 Crowd Campaign: 31 unique clicks generated 11 reviews, a 35% rate.
  • Non-incentivized G2 Crowd Campaign: 13 unique clicks generated 6 reviews, a 46% rate.
Insight:

Surprisingly, our non-incentivized emails produced the best conversion rate of online reviews. While that campaign had the lowest open and click through rate percentage, a near majority of customers who did click the link generated a review, without being incentivized.

Online Review Sentiment of the Campaigns

Capterra Campaign generated:

  • 13 Reviews with 5 Stars
Incentivized G2 Crowd Campaign generated:

  • 8 Reviews with 5 Stars
  • 2 Reviews with 4.5 Stars
  • 1 Review with 4 Stars
Incentivized G2 Crowd Campaign generated:

  • 5 Reviews with 5 Stars
  • 1 Review with 4.5 Stars
  • 1 Review with 4 Stars
Insights:

Correspondences coming directly from Grade.us produced 5 star ratings across the board, whereas the emails coming from G2 Crowd produced a few 4 and 4.5 star reviews. It makes sense that customers leaving a review because we asked them to resulted in more positive reviews.

You could also make the argument that incentivized reviews did not necessarily produce biased results. If the incentives had influenced the star rating, we most likely would have seen 5 star reviews across the board like our internal email campaign.

Takeaways


Every business and every online review site are different, and there are tons of additional variables the make it difficult to prove causation in this experiment. But that doesn’t mean we cannot look at the results and pull some meaningful insights.

If our client’s service is subscription based, an email blast campaign is an effective strategy for reaching out to our customers all at once.

Using incentivizes in our review acquisition email campaigns seemed to produce the most reviews, however it’s clear that when the ‘Ask’ comes from the actual company, the results are even better.

Customers are more likely to open an email with an incentive in the subject line, but that doesn’t mean that they’re more likely to actually write the review.

Finally, remember that not all Online Review sites like Capterra or G2 Crowd allow for incentivizing reviews. The fear tends to be that incentivizing reviews will influence the authenticity and credibility of the reviews. We found in our experiment, that wasn’t the case, however, incentivizing review acquisition still has its risks.

About the Author

Garrett Sussman

Twitter

After 6 years of San Francisco living, Garrett has returned to his roots in New Jersey to manage Content Marketing at Grade.us. He infuses his writing with humor, authenticity, and thoughtfulness. He also happens to be a fan of graphic novels, Philadelphia sports, and content marketing.