Our PPC team conducted a comprehensive experiment to determine whether AI-generated Google Search Ads can outperform their human-written counterparts. Our aim was to compare the performance of ChatGPT's AI-written ads against the ads crafted by our ESC PPC team.

We carefully designed the experiment across three distinct ad groups, each with two ads, to ensure reliable results and valuable insights. We focused on Dynamic Search Ads, one written by our PPC team (ESC Copy) and one generated by ChatGPT (AI Copy) tailored to each ad group theme.

We set the ad rotation in each ad group to "Do not optimise: Rotate ads indefinitely." By rotating the ads without any optimisation bias, we aimed to give every ad variant an equal opportunity to showcase its potential.

The criteria for success were straightforward - we considered an ad to outperform the other if it generated more conversions, achieved a higher conversion value, and delivered a better ROAS while staying within budget.

Excitingly, we're here to share the intriguing findings, which might reshape the way we approach advertising strategies.

Will the AI-generated ads prove to be more effective, or will the human-written ads retain their edge?


Let’s work together to build your store

We are one of the world’s most trusted and experienced Shopify Plus Partners.

Get in touch

Experiment Results

Let's explore the findings and see which ad variant emerged victorious after running this experiment and the ads for a total of 72 days.

Let’s work together to build your store

We are one of the world’s most trusted and experienced Shopify Plus Partners.

Get in touch

Ad Group 1

The results of ad group 1 provided the most interesting insights in our experiment. The results showed that the ESC ads, despite spending 33% more, received 43% fewer impressions and 29% fewer clicks compared to the AI ads. This led to a 88% higher CPC. However, the ESC ads resulted in a 23% higher CTR. Most notably, the ESC ads outperformed the AI ads in terms of conversions, achieving an impressive 298% more conversions, leading to a remarkable 462% higher conversion rate and a significantly lower cost per conversion (66% lower). Moreover, the ESC ads generated an astounding 478% more conversion value and achieved a 334% higher ROAS.

Ad Group 1

Metrics AI Ad Copy ESC Ad Copy Difference to AI Copy
Cost £145.92 £194.40 33%
Impr 3,702 2123 -43%
Clicks 253 179 -29%
CPC £0.58 £1.09 88%
CTR 6.83% 8.43% 23%
Conv 5 22 298%
Conv rate 2.16% 12.15% 462%
Cost/conv £26.68 £8.94 -66%
Conv val £172 £994 478%
ROAS 118% 511% 334%

Let’s work together to build your store

We are one of the world’s most trusted and experienced Shopify Plus Partners.

Get in touch

Ad Group 2

Ad Group 2 revealed some intriguing results. The ESC ads, compared to the AI ads, showed a significant difference in spending, allocating 263% more budget. As a result, they received 222% more impressions and 285% more clicks. Interestingly, the ESC ads achieved a 6% lower cost per click (CPC) and a 19% higher click-through rate (CTR).

The ESC ad achieved 165% more conversions than the AI ad. Despite this, the conversion rate was 31% lower for the ESC ads. Moreover, the cost per conversion was 37% higher for the ESC ads, which is worth noting.

The ESC ads managed to generate a substantial 360% more conversion value, contributing to their 27% higher return on ad spend (ROAS).

Ad Group 2

Metrics AI Ad Copy ESC Ad Copy Difference to AI Copy
Cost £63.88 £231.91 263%
Impr 1,422 4,585 222%
Clicks 87 335 285%
CPC £0.73 £0.69 -6%
CTR 6.12% 7.31% 19%
Conv 9 23 165%
Conv rate 9.99% 6.87% -31%
Cost/conv £7.35 £10.08 37%
Conv val £294 £1,356 360%
ROAS 461% 585% 27%

To sum it up, in Ad Group 2, the ESC ads clearly outperformed the AI ads in terms of conversions, conversion value, and ROAS, despite some differences in click-related metrics and cost per conversion.


Let’s work together to build your store

We are one of the world’s most trusted and experienced Shopify Plus Partners.

Get in touch

Ad Group 3

In Ad Group 3, the ESC ad spent 56% more, resulting in 43% more impressions and 49% more clicks compared to the AI ad. While its cost per click (CPC) was 5% higher, it managed to secure a 4% higher click-through rate (CTR).

The true standout performance of the ESC ad in Ad Group 3 was evident in its conversions. It achieved an impressive 208% more conversions than the AI ad, leading to a remarkable 107% higher conversion rate. Moreover, the ESC ad significantly lowered their cost per conversion by 49%, which is quite noteworthy.

As a result of their higher number of conversions, the ESC ads generated 167% more conversion value, contributing to their 71% higher return on ad spend (ROAS).

Ad Group 3

Metrics AI Ad Copy ESC Ad Copy Difference to AI Copy
Cost £94.76 £148.05 56%
Impr 4,022 5,752 43%
Clicks 171 254 49%
CPC £0.55 £0.58 5%
CTR 4.25% 4.42% 4%
Conv 5 15 208%
Conv rate 2.92% 6.06% 107%
Cost/conv £18.95 £9.63 -49%
Conv val £105 £281 167%
ROAS 111% 190% 71%

To sum it up, in Ad Group 3, the ESC ads clearly outperformed the AI ads in terms of conversions, conversion value, and ROAS. Their increased budget investment paid off with higher impressions and clicks, further solidifying their superior performance. Ad Groups Combined

Taking all ad groups into account, the ESC ads showed some impressive results. They spent 89% more and, as a result, received 36% more impressions and 50% more clicks compared to the AI ads. Although their cost per click (CPC) was 25% higher, they managed to achieve a 10% higher click-through rate (CTR).

Now, let's talk about conversions – the ESC ads really nailed it! They secured a remarkable 214% more conversions than the AI ads, leading to an outstanding 109% higher conversion rate. Additionally, the ESC ads had significantly lower cost per conversion (40% lower compared to the AI ads), making a substantial impact.

When it comes to generating value, the ESC ads didn't disappoint. They scored a whopping 360% more conversion value, contributing to their impressive 144% higher return on ad spend (ROAS).

Ad Groups Combined

Metrics AI Ad Copy ESC Ad Copy Difference to AI Copy
Cost £305 £574 89%
Impr 9,146 12,460 36%
Clicks 511 768 50%
CPC £0.60 £0.75 25%
CTR 5.59% £6.16% 10%
Conv 19 60 214%
Conv rate 3.75% 7.83% 109%
Cost/conv £15.89 £9.55 -40%
Conv val £572 £2,631 360%
ROAS 188% 458% 144%

In summary, across all ad groups, our ESC ads clearly outperformed the AI ads in terms of conversions, conversion value, and ROAS. Their increased spending was justified with higher impressions and clicks, demonstrating their overall better performance.

Here are the metrics we watched most closely visualised:

Overall, the ESC ads generated 60 conversions and a conversion value of £2,631, while the AI ads generated 19 conversions and a revenue of £572.


graph showing conversions and value of AI vs human ad copy. human shows better results.


The ESC ads achieved a combined ROAS of 458%, while the AI ads achieved a disappointing combined ROAS of 188%.


graph showing return on ad spend of AI written ad copy vs human ad copy. AI is 188% and human is 458%


Let’s work together to build your store

We are one of the world’s most trusted and experienced Shopify Plus Partners.

Get in touch

Conclusion

In this experiment, our human-written ads emerged as the champions, outperforming AI-generated ads. Their artful creativity led to stronger engagement, higher conversions, and cost-efficiency, showcasing the unique strengths of human-crafted content in advertising. The human touch remains an irreplaceable force in driving impactful campaigns and forging genuine connections with the audience.

As we continue to explore the vast potential of AI and its integration into advertising strategies, we must acknowledge and cherish the extraordinary capabilities that human-written ads bring to the table. In this dynamic landscape, the partnership of AI-driven insights and human creativity holds the key to crafting ads that truly leave a lasting impact on consumers and propel brands to new heights of success.

To stay up to date with all things ecommerce, make sure you subscribe using the form below to receive our weekly newsletters, guides and advice.