AI vs. human study:Can consumers tell the difference between AI and human-generated content?

Unless you’re living under a rock, there’s no doubt you’ve heard about artificial intelligence (AI) and its effect on society as we know it. Tools like ChatGPT, Copy.ai, and Dall-E are revolutionizing how people work, create, and live their everyday lives.

Businesses around the globe are examining ways they can use AI to increase productivity, capitalize on the market share they didn’t initially think possible, and improve overall operations. But with pioneering any new technology, AI has drawbacks.

The rising popularity of these new tools has also been a cause for concern. Many prominent tech and business giants like Steve Wozniak, Elon Musk, and Warren Buffet are concerned with the trajectory of AI platforms — so much so that these three titans of industry attached their signatures to an open letter calling for all AI companies to pause development for six months.

Other prominent business figures have followed suit and expressed their concerns surrounding AI. Technology minds like Musk are concerned about where AI can go if this new technology is left unchecked and evolves without regulation for much longer. Additionally, certain applications like ChatGPT and Copy.ai raise other concerns around ethics and plagiarism.

Still, other businesses are unsure how AI will affect their key performance indicators (KPIs). The effects this technology will have on factors like sales, user experience, and customer retention still remain unclear. It’s too early in AI’s evolution to gauge customer sentiment of the technology. 

These reasons are among the many that prompted our team at Nexcess to launch an in-depth study into how consumers perceive AI. We’ll dive into the details and results of the study down below, but our hope is that covering this topic will give all parties involved a better understanding of the current state of the technology, as well as its trajectory to come.

What is artificial intelligence (AI)?

Sometimes, you can hear a term so often in the news and social media that its true meaning can get lost. Therefore, before we dive in, we should tackle the basic question, “What is artificial intelligence?”

Artificial intelligence is using machines and computers to simulate the intelligence of human beings. The goal is to optimize performance and improve problem-solving across industries by combining the efficiency of a computer with the creativity and rationalization of the human brain.

Under the “artificial intelligence” umbrella are more specific technologies like speech recognition, natural language processing, and machine vision. These innovative use cases are all designed to bring AI one step closer to processing the world with the same capability as a human being. But, at this stage in its evolution, AI still has some limitations.

AI writing tools like ChatGPT and Copy.ai have come a long way. They provide knowledge on a wide range of topics in a conversational, blog-style format that saves many business owners time and energy in their own marketing efforts. But these platforms have been known to fabricate data and write in confusing, unorthodox language.

In addition, AI writing platforms are still learning the “human” element. When their information is solid, it can sometimes be written in a cold, mechanical style.

Visual AI platforms also have limitations on what they can produce. While some of the images these platforms come up with are amazing, they can struggle when it comes to producing images with human subjects. 

Occasionally, these platforms also have issues with images containing text. AI platforms will generate the images, but the text within them will be false or fabricated. Businesses can use these platforms to create logos or brand artwork without a problem, but if you need any portrait of a human being or anything more complex, you may be out of luck at this point.

The question now becomes, how deep do these drawbacks go, and how do they play into AI’s advancement as a whole? Our survey of 1,000 participants hopes to answer just that.

Details of the study

These questions regarding AI’s efficacy are what our study is all about. Survey participants from various age groups were shown sets of ecommerce images and copy. These sets consisted of an AI-generated element and a human-generated element side-by-side, based on identical prompts. 

Questionnaires were administered only to those participants who claimed to be moderately familiar with AI. These users were broken down into the following demographic categories:

Respondents were shown the image and copy sets and asked to select which one they thought was generated by AI. Since one of AI’s current drawbacks is generating pictures of humans, we made sure not to use human images as part of our study since they would be too easily identifiable as AI-generated. 

Now that you’re familiar with the survey’s framework let’s dive into what we learned. 

Want to test your skills? Take the same survey as our respondents did below.

1. Respondents could identify the AI-generated content approximately 54.64 percent of the time

Across all demographic groups, respondents could identify the AI-generated content about 54.64 percent of the time. Respondents seemed to struggle with our set of bicycle images. Most participants (54.6 percent) selected the orange bicycle as AI-generated when it was actually our human-generated option.

Moving on from there, however, the respondents performed much better. They correctly guessed the bicycle call-to-action, email copy, makeup bag, and social post. The only other survey elements that seemed to trip them up were the refrigerator and shampoo images — with 54.3 percent and 51 percent mistaking the human image for AI, respectively.

2. Younger respondents could more accurately identify AI-generated content

One interesting trend emerging from our survey is that younger respondents seemed to be better at distinguishing AI from human elements. Participants in the 18-24 age bracket answered correctly 61.29 percent of the time. That ratio dropped to 55.75 percent for people aged 25-34, falling further in the 35-44 bracket with a 53.49 percent accuracy ratio. As age increased to the 45-54-year-old bracket, accuracy again dropped to 50.89 percent.

The oldest participants, the 54+ bracket, were unique outliers to our survey. Their accuracy ratio bucked the trend at 52.63 percent. However, this may have to do with the number of participants, as there were significantly fewer respondents in the 18-24 group than in the 54+ bracket. 

Another study conducted by the Baycrest Center for Geriatric Care may give some insight into why the older population struggles with identifying AI content. While the study had no concrete evidence of the reasons behind this trend, they did discover the same trend as our survey.

Researchers in the study analyzed their data and came to the conclusion that as we get older, we focus on different aspects of what we hear or see. For example, rather than processing the picture as a whole, a more senior participant may focus on certain image features that make the picture appear more real.

Concerning audio generated by AI, individuals tend to zero in on the words someone is saying as they get older – rather than focusing on tone or cadence of speech, which can be telltale signs of non-human audio.

3. AI-generated copy was more likely to be correctly identified as AI-generated content than images

It was easier for our participants to spot AI-generated copy (57.3 percent accuracy) than AI-generated images (53.36 percent accuracy). This may have to do with the tone and structure inherent to AI writing.

There’s something to be said for direct, no-fluff copy when it comes to content creation. But AI tools seem to take that a bit too far. When looking at a passage of AI-generated copy, it’s relatively easy to identify their cold, mechanical languages. They also tend to use awkward phrasing, almost like they’re trying too hard to be human.

Spotting AI copy easily may begin to affect the success of ecommerce businesses. If a potential customer reads a product description and flags it as AI copy, it may turn them off toward your brand or product, resulting in a lost sale. Additionally, copy with the human element provides the extra emotion and persuasion necessary to convert a potential customer from ‘warm lead” to “sale.” Subsequently, human-generated content may not only keep the customers you have but help you win new ones as well.

4. AI-generated images were harder to detect than AI-generated copy

Respondents of our survey also found it easier to identify the AI copy than AI images. The short, blunt writing style of AI bots may be easier for individuals to spot. It has less emotive language and is often overtly to the point — in fact, it’s “directness to a fault.”

On the other hand, it was more difficult for survey participants to accurately select the AI images. Over half (51 percent) of survey-takers thought the human-generated shampoo bottle image was created by AI and 53.4 percent mistook the human-generated refrigerator image for AI, as well. 

What does this mean? It could be a combination of things. AI platforms may be evolving more rapidly in the visual sector vs. the copy sector. But it could also mean that differences in images are harder to detect than differences in tone, voice, and sentence structure. Either way, this point does demonstrate the power of AI if you’re in need of generating images, like for ecommerce products or business logos.

AI vs. human-generated content study: Looking forward

Although the results of our study were relatively close, the overarching trend is that businesses shouldn’t ignore AI. If used properly, AI tools can be instrumental in optimizing your content production and improving the performance of your ecommerce store. Just be sure to review and apply the “human touch” to anything you ask AI to generate for you.  

Overall, it’s hard to tell what the future holds for AI. Popular platforms like ChatGPT are undergoing evolutions that allow them to execute trillions of functions. Also, AI as a whole continues to move toward a concept known as Artificial General Intelligence, which some experts believe to be the “pinnacle” of AI’s human-like abilities.

As for where AI is actually headed, it’s still too soon to know. But one thing is for certain. This technology is a powerful tool and learning how to use it, and use it properly, can only serve you and your business as we approach a new technological frontier.

Want to try for yourself? Take the test