A/B Testing Online Review Incentives Via Email Campaigns

In the past we’ve discussed the risks and rewards of incentivizing online reviews. While Yelp is notoriously strict and believes (wrongly) that even asking for a review biases it, other review site publishers are more, um, rational. Sites that have a predominantly professional audience recognize that their reviewers’ time is valuable. Providing a small reward for their time can actually motivate more unbiased reviewers to participate. But it has to be done the right way.

Last week, we collaborated with the awesome B2B software review site G2 Crowd on an email campaign to get our latest crop of Grade.us customers participating in writing reviews. Because honestly, if an online review marketing and management platform lacks online reviews, then how can you trust their value proposition? 🙂

G2 Crowd actually encourages and facilitates providing reviewers a small reward to review products like ours. With review incentives being a hot topic–and one that’s misunderstood or over-simplified by both proponents and detractors–we saw an opportunity to A/B test the effectiveness of incentives on online review acquisition.

We had a few questions that we wanted to answer:

We frequently talk about how it’s important to have a stream of new online reviews for our businesses, yet not all business models make this strategy easy. Velocity of online reviews, does play a role as a ranking factor for Google.

If our business model consists of one time transactions, based on products or projects, we can trigger email drip campaigns or SMS text campaigns for getting quality reviews at the time of sale or at the completion of the project. But if our services come on retainer, or in the case of Grade.us, an ongoing subscription model, finding the right time to ask for the review can be a bit more tricky.

Using email campaigns is an effective marketing tactic, one of many recommended by WeeklyM, a marketing strategy website. While it’s not as ideal as a steady stream of reviews, we’ve found value in sending out email blasts to a portion of our customers at a time.

Prior to our G2 Crowd experiment, we sent out our own email blast to a selection of new customers, to bolster our Capterra.com listing.

At the bottom of our Product Update newsletter, we dedicated the last section of the email as a dedicated call-to-action for new Capterra reviews. Here’s the copy we used:

Here we go… dogfood time: You know how much customer reviews mean to us, right? Reviews are our business, after all. But for me, the value goes way beyond that.

I literally choke up when I read the kind things our customers say about Grade.us. And the critical things? They keep me up every night.

We’ve had some growing pains the last couple months, and as a result, I watched exactly zero March Madness games 🙁

But now one of our favorite B2B review sites, Capterra, is running its “Reviews Madness” promotion. And I can tell you that it would probably move me to tears if you’d write a review of Grade.us during this competition. It can be super quick–Capterra is easy. No signup. No fuss.

Seriously, I’d be forever in your debt–as would all those other professionals looking for a good review management solution on Capterra!

In the copy, our founder Jon Hall, appealed to our customer’s emotions. We’re not a massive organization, so it was essential for us to connect with our loyal customers in a personal way. Jon’s genuine words tugged at the heartstrings of our valued customers, and the results were beautiful.

Here are the results:

Our 67.4% open rate is highly favorable compared to industry benchmarks provided by MailChimp. Industry benchmarks can be extremely helpful in comparing the success of our campaigns, but there’s just as much value by comparing results internally to past campaigns.

With the same ultimate goal as our email campaign to earn Capterra Reviews, our email campaign experiment with G2 Crowd highlights both incentivized versus non-incentivized email campaigns, but also the difference in email campaigns coming from our company versus a third party.

Incentivized reviews are still polarizing, but most often it’s taken in the wrong context. Isn’t there a difference between incentivizing a positive review versus using incentives to simply earn a review regardless of its sentiment?

In the context of this experiment, we felt it was valuable to reach out to G2 Crowd to get a better understanding of this critical difference:

For a business that is built on the authenticity of online reviews, we can imagine that G2 Crowd takes the side-effects of incentivizing online reviews seriously. They’ve done the research to back it up.

The email campaigns were sent out on July 27th. G2 Crowd sent out 161 incentivized emails and 161 non-incentivized on our behalf.

G2 Crowd kept the email copy simple, and we went in with a few adjustments so that the message stayed ‘on-brand.’

For the incentivized customers, we offered a $25 Amazon gift card through G2 Crowd.

Hi *First Name*!

I’m reaching out in the hopes you could contribute a review of Grade.us on G2 Crowd. Grade.us asked us to contact you and its other customers to obtain unbiased, authentic reviews of its product.

To say thanks for taking the time to share your feedback, we’re offering a $25 Amazon.com gift card* to the next 20 Grade.us users who complete a review. The gift card is from G2 Crowd and your eligibility to receive it is in no way conditioned on the content of your review.

You can add your review of Grade.us here.

If you have any questions about G2 Crowd or the review process, don’t hesitate to ask.

Thanks for your help,

Molly

And here’s the email we sent out without an incentive:

Hi *First Name*!

I’m reaching out in the hopes you could contribute a review of Grade.us on G2 Crowd. Grade.us asked us to contact you and its other customers to obtain unbiased, authentic reviews of its product.

You can add your review of Grade.us here.

If you have any questions about G2 Crowd or the review process, don’t hesitate to ask.

Thanks for your help,

Molly

The email campaign generated a total of 17 reviews, which exceeded our expectations, knowing how difficult it can be to actually get the reviews.

Reviews generated: 6

Reviews generated: 11

We found some interesting results from this experiment.

Emails originating from Grade.us were opened more than the campaigns from G2 Crowd. Our Capterra campaign saw an open rate of 67.4%, whereas the G2 Crowd non-incentivized and incentivized emails had an open rate of 41.7% and 54.9% respectively.

It makes sense that a personal email from the brand would generate more opens than a third party email. Additionally, including the “Get an Amazon Gift Card” in the subject line had a positive impact on opens.

Despite being a platform feature update, our Capterra email campaign produced a welcomed 15.8% click through rate. The G2 Crowd non-incentivized and incentivized emails produced a click through rate of 8.6% and 20.3% respectively.

Incentives seemed to have the strongest impact on click through rate. While the Grade.us email had the highest open rate, the incentivized emails produced the highest click through rate. It appears that the Amazon Gift card offer helped get more customers over to our G2 Crowd listing.

Here’s where it gets very interesting:

Surprisingly, our non-incentivized emails produced the best conversion rate of online reviews. While that campaign had the lowest open and click through rate percentage, a near majority of customers who did click the link generated a review, without being incentivized.

Correspondences coming directly from Grade.us produced 5 star ratings across the board, whereas the emails coming from G2 Crowd produced a few 4 and 4.5 star reviews. It makes sense that customers leaving a review because we asked them to resulted in more positive reviews.

You could also make the argument that incentivized reviews did not necessarily produce biased results. If the incentives had influenced the star rating, we most likely would have seen 5 star reviews across the board like our internal email campaign.

Every business and every online review site are different, and there are tons of additional variables the make it difficult to prove causation in this experiment. But that doesn’t mean we cannot look at the results and pull some meaningful insights.

If our client’s service is subscription based, an email blast campaign is an effective strategy for reaching out to our customers all at once.

Using incentivizes in our review acquisition email campaigns seemed to produce the most reviews, however it’s clear that when the ‘Ask’ comes from the actual company, the results are even better.

Customers are more likely to open an email with an incentive in the subject line, but that doesn’t mean that they’re more likely to actually write the review.

Finally, remember that not all Online Review sites like Capterra or G2 Crowd allow for incentivizing reviews. The fear tends to be that incentivizing reviews will influence the authenticity and credibility of the reviews. We found in our experiment, that wasn’t the case, however, incentivizing review acquisition still has its risks.