March 2, 2023

ChatGPT and The Challenges for Search Accuracy

Accuracy is a broad topic and there are many directions this discussion could take. There have been many reports, aside from the astounding capability to integrate static web content (from web crawls) into cohesively written responses versus the long list of findings that arises from a traditional search.

However, there are also reports of inaccurate and misleading responses. I want to focus not on these specific instances but on the overall challenge ChatGPT or those products that use its technology for search face in maintaining high accuracy on all responses.

Searching with ChatGPT’s Current Data

The trial ChatGPT version available through OpenAI was trained on static data that could be today as old as 2 years. Thus, it’s not surprising that when asked about current events it provides aged information. Microsoft within Bing is modernizing the information used in its interactions but as of today remains a work in progress. Microsoft likely will use Prometheus to real-time train the model on queries where the information is missing, aged, or recently updated.

It is instructive to lay out how an AI engine like ChatGPT or any of its partners using the underlying technology will remain current. When it was put together ChatGPT had the luxury of scanning the web, storing the information for testing purposes, and not dealing with the real-time updating requirement.

Included sources were Wikipedia, some 25 GB of books, 100 GB of journals, Reddit links, and 570 GB of web crawl. So whenever you ask ChatGPT a question it draws from this static set of information. The construction of an answer is gleaned not only from the content but how it learned to construct responses. This is truly amazing part of the offering.

Can Information Be Kept 100% Current?

Those sources of information are of course not static in all cases and there is no defined mechanism for any user of information to receive notification of change. When I was in marketing many years ago, we prepared sales to slide decks for our products.

We would update them from time to time as changes were needed. But when I would travel with sales, I would always find a case where the sales rep was using an old deck despite instructions to keep it current and outbound messages to alert them when a new deck was available. This was a manual process in a constrained environment.

And as such, it was impossible to keep 100% current. Scanning the web is incalculably more difficult despite the processing power of the cloud. In ChatGPT’s case, there are today 1.5-year (see previous web link) intervals between releases of its newly trained versions.

Implementing ChatGPT with Microsoft’s Bing Search

Now let’s take it to the search area where Microsoft is planning to use the technology to complement Bing search. Early versions of Bing showed traditional search results, image results, video results, and chat results as separate tabs.

If you are going to open queries to anything the user wants to ask and demand accuracy, training must be a continuous exercise or the system must be smart enough to map out where to go once information sources are identified. And bring those sources in for real-time training all within the expected response time by the questioner.

Pre-training by Microsoft would seem to be impossible. To use an astronomy analogy that helps people understand the vastness of the universe, Bing will likely have to interact with as many information sources “as there are grains of sand on the earth”. So, it’s likely that it would use its search capabilities to look at maybe the first few pages of a traditional search and train the model on what it found. Still seems daunting.

The extent of the real-time model training of the model must be bounded by the expected response time of the inquirer. Achieving perfect accuracy for any questions asked seems to be one of the most difficult challenges Microsoft has ever faced.

Bottom Line

Expecting the results of any generic search to generate ChatGPT-like output with a high degree of accuracy is an extreme technical challenge. Evidence shows that Microsoft and others will simply provide such results alongside traditional search results leaving it to the user to verify accuracy.

The best results in combining search and ChatGPT’s language model will be in walled garden applications where the training data set is either static or experiences minimal changes.

Tune into our LIVE Keynote Session with our Analysts on ChatGPT!

Join us for our annual Transform Tour 2023 on Wednesday, March 22, 2023 at 10 AM PT / 1 PM ET – click here to RSVP.

ChatGPT and The Challenges for Search Accuracy