Can ChatGPT create factually accurate documents for practical use?

James Kenny

Head of Programmatic and Paid Social


Industry News

One of the main questions people ask is if you can use AI tools for purposeful reasons. Our Creative team has tested ChatGPT to see how it fared with content production and delivered eye-opening results. A concern is that while many people around the globe use ChatGPT, it is often more for ‘fun’ than serious reasons, an opinion confirmed in a 2023 Statista study about use cases for generative AI.

It may be the best (and possibly only) way to use ChatGPT, but our Programmatic team thought differently. The team tasked ChatGPT with something more practical: creating a document to help companies set up a business account on Facebook. Much like the Creative team, the results our Programmatic specialists saw were reassuring but also surprising.

What were the main reasons the team chose to use ChatGPT?

 

ChatGPT can create content very fast

 

Although humans can type quickly, ChatGPT is faster and, for the most part, can create content much quicker than humans. If the situation were that a client urgently needed a document (which is what our Programmatic team based this experiment on), ChatGPT would be the more time-saving choice.

 

ChatGPT has the best reputation

 

Since ChatGPT’s launch in November 2022, other generative AI tools have appeared, including Google Bard and Bing Chat by Microsoft. Yet, according to a poll of over 2,000 individuals in the US, ChatGPT is still the most widely used and trusted of all generative AI tools, which was the team’s reasoning for using it to create the document.

 

 

What prompt did the Programmatic team give ChatGPT?

 

As knowledge increases about how to get the best results from AI, tools like ChatGPT are only as good as the information you enter into them. An advantage to the programmatic team prompting ChatGPT is they know the process for creating a Facebook business manager account inside out. The prompt our specialists gave ChatGPT was detailed enough for it to interpret the information, and our team was able to spot any glaring omissions about the process in the document ChatGPT created. The prompt said:

Hello. Can you please write me a guide based on the following requirements:

Background: I am working for an agency for a new client. The client needs to setup a new Facebook Business Manager, A New Ad Account, A New Pixel and then to share access with us.

Requirements: This needs to be client-facing, so from the point of view of the client. It’s unclear what technical capacity they have. The client did have an old Business Manager, but the old Admin created it and left, so no-one has experience in setting something, but we do need to build security, and future safeguarding into consideration (to ensure that they don’t lose access if the current Admin leaves).

Did ChatGPT create the document correctly?

 

As expected, ChatGPT created the document our Programmatic specialists requested to set up a Facebook business account. The document was formatted into digestible steps and can be easily interpreted by new or experienced users.

ChatGPT took around 10 minutes to create the document rather than 30 minutes, which was the time the team speculated it would have taken them to make the same document, confirming ChatGPT’s time efficiency. But while ChatGPT created the document quickly, the team still needed to extensively check it to make sure what ChatGPT included was factually accurate.

As the team knows each step involved in creating a Facebook business account, the fact-checking took little time and did not cause any problems. However, problems may have arisen if the team were checking the accuracy of a process they knew little about because they would need to know if what ChatGPT produced was incorrect and then amend any mistakes. It’s advisable to have an in-depth understanding of what you are asking ChatGPT to get the best results from using it.

 

 

ChatGPT (like all generative AI) is a brilliant tool to help businesses become more time-efficient, especially in a reactive industry such as digital marketing. As our Programmatic team found out, along with 31% of other technology professionals around the globe, ChatGPT can create documents for practical use. Yet, there still needs to be a strong level of human oversight to ensure the outcomes are effective, accurate, and successful.

Interested to learn more about where AI may take us?Download our whitepaper 'The Future of Digital'