It’s been 15 months since ChatGPT’s groundbreaking debut and about one year since Google Bard (rebranded as Gemini on Feb. 8) joined the GenAI fray. At this point, it feels like every marketer and communications professional on the planet has experimented with, if not regularly used, this emerging technology. After all, much of the job is researching, ideating, proofreading, drafting, fact-checking, and fine-tuning SEO strategy—all things with which GenAI can lend a hand.
But which is superior for marketers and communications professionals?
Over the past several weeks, we put GPTv3.5 and Gemini (and its predecessor Bard) through their paces, asking each to complete a quiz designed to test its comms savvy. Note: we did not look at any image generation capability as that is outside the scope of the free versions of the models. Google has also faced major criticism for the way Gemini handles race in its image generation and in some responses, which also isn’t covered by this head-to-head comparison.
We approached the testing with no strong bias or preference. But it is hard to ignore the fact that Bard/Gemini was capable of accessing the internet from day one thanks to its integration with Google’s search technology. Meanwhile, the launch version of ChatGPT was limited to the knowledge it was fed during training. Since that time, OpenAI has enhanced ChatGPT through its integration with Microsoft Bing.
Most of our queries were objective, focusing on topics that our PR and digital teams are naturally fluent in—like paid search/SEO, paid social, PR, research, and content creation. Others, which were designed to test the tools’ speed and ability to pull relevant insights when researching, were more subjective.
The Findings
While neither OpenAI nor Google’s LLM is perfect (that goes for both versions of Google’s products; more on that later), both have a lot to offer. I will say this about both tools – whether intentional or not – both demonstrated a sense of humor, often getting cheeky with their outputs.
During our initial test, we asked ChatGPT to tell us about Bard. It’s response? “Google Bard is a hypothetical project or concept, and there’s no information available… unless it was released after January 2022,” which, of course, it was. When we asked Bard the same question,it replied, “You’ve already asked me who I am a couple of times today, and I’m happy to answer again!” before explaining its features.
Jokes aside, both LLMs performed well on the test overall. Let’s break it down:
We ran our queries at different times of day and night and, fortunately, we never encountered any issues accessing either service. This was a welcome change, as OpenAI’s servers frequently encountered capacity issues in ChatGPT’s early days. It appears their work to expand capacity paid off, as ChatGPT consistently beat Gemini/Bard in the race for a response.
More often than not, GPT began answering our queries almost immediately and delivered a complete answer in as few as three seconds. If the tools were actually intelligent, we’d say Gemini seemed more contemplative, appearing to give answers actual thought before spitting out its answer. This is likely due to Gemini’s access to up-to-date reference material which includes the active internet and Google’s vast knowledge graph in addition to its trained data. ChatGPT relies solely on its training data, meaning there is no “search” before it responds.
That said, Gemini was no slow poke. While seemingly slower than ChatGPT, it was more than fast enough, in our opinion, to answer most queries in under eight seconds. The slowest response time by both chatbots was 14 seconds. However, this was related to a rather detailed query about developing Facebook ads and targeting a subset of LinkedIn users with paid content, so the additional time was to be expected.
In terms of accuracy and depth, both bots did well and rarely returned incorrect answers. Of note, there was no tangible difference in performance between Bard and Gemini in terms of output. We assumed that, with the rebrand of the tool, Google would adjust or improve its model, but that doesn’t seem to be the case. However, Gemini/Bard consistently returned more nuanced and detailed responses, while ChatGPT, at times, provided the bare minimum. .This is almost certainly a product of their differing reference sets and response programming. To this day, ChatGPT qualifies its answers with “as of my last update in January 2022…”— an all-too-common and frustrating occurrence for any regular user.
Drilling in a bit on the results returned, a question about a specific broadcast journalist yielded a non-answer from ChatGPT (see paragraph above about “last updates”) but Bard gave a very accurate and thorough answer. Interestingly, Gemini’s answer a few weeks later to that same question was far less helpful, but we considered that an outlier. Gemini also gave a far more detailed answer about AI networking groups. In terms of social media and paid social media questions, the answers from both chatbots were nearly identical. In the SEO realm, Gemini pushed back a bit on the query, asking about goals, target audience and their consumption habits versus simply spitting out its answer.
The Verdict
Though both tools performed well, the overall winner, in our view, was Gemini! Its answers were richer and more detailed, and despite its slightly slower processing, the depth and timeliness of its responses were worth the small wait. We found that its additional features – like the one-click “double check with Google” option – made it more pleasant to use.
At the end of the day, the race between these two tools is just beginning—and there’s certainly more progress to come. We’ll be watching closely to see how our opinions change, and if another reigning AI champ rises in the coming months.
What has your experience been with Gemini and ChatGPT? Do you have a differing opinion on which of these two GenAI marvels is superior when it comes to its value in the marketing and PR tech stack?