Generative AI is handy for condensing and simplifying large amounts of information. Use cases for marketers and business leaders include:
- Explain complicated documents, such as patents.
- Summarize long stories into key points and definitions.
- Extract important sections from ebooks and white papers for blogs and promotional materials.
The following are four generational AI tools to accomplish these tasks. To test, I asked the tools to analyze a 2023 patent from Google for Search Generative Experience, with three prompts:
- What is this document about ?
- How does SGE select external resources in its responses? Do the resources come from organic search results?
- Define the essential terms of the document.
My goals were to assess the accuracy of the answers and check for “mind-blowing” – inventive – information.
ChatGPT
ChatGPT can now process uploaded documents, including PDF files. It provided a generic overview of the question “What is this document about?” »
But it was the only tool that answered the second question: “Did the assets come from organic search results?”
According to the patent document, not all documents are selected from organic search results alone. The selection process involves different categories of documents resulting from the search:
Query-responsive search results documents: These are documents obtained in response to the original query submitted by the user.
Search result documents responsive to related queries: These documents come from searches carried out for queries related to the original query.
Research results documents responsive to recent research: These are documents that responded to recent queries, possibly related to the current query or the user’s search history.
ChatGPT has composed the category names (in bold)! The patent does not use these terms.
So I responded, “I don’t see these category names in the original document. Did you invent them?
Response from ChatGPT:
You are correct and I apologize for the confusion. The category names I used (“Query Responsive Search Results Documents”, “Related Query Responsive Search Results Documents”, and “Recent Search Responsive Search Results Documents”) were indeed my own wording to summarize the concepts described in the patent. They are not directly quoted from the document.
The patent actually deals with the process of selecting documents based on their responsiveness to a given query, related queries, and recent searches. These selections can include documents that answered the original query, as well as those responding to related or recent queries. My intention was to clarify this process, but I should have made it clear that these category names were my own summaries and not direct quotes from the document.
Thank you for pointing out this discrepancy.
ChatGPT’s answer is an example of hallucination, a weakness of many AI generation tools.
Claude
Claude is an independent generative AI model from Anthropic that does not use ChatGPT, unlike the other tools on this list. During my tests, Claude was the most cautious in his response.
For example, the other three tools answered “Yes” to the question “Are documents selected from organic search results?” Only Claude offered a nuanced response:
The patent does not explicitly state whether documents are selected from organic search results.
However, a few contextual clues suggest that they refer to organic search results:
-
- The background describes providing query-based summaries of “search results documents” without specifying whether these are organic or paid results. Typically, “search results” refer to organic results returned by a search engine.
- There is no mention of filtering or treating paid/promoted results documents differently from organic documents when selecting documents for summarization.
- The goal appears to be to summarize the documents most relevant to the search query in order to provide an overview to users. Organic search results are generally more optimized for relevance than promotional or paid results.
- Related queries are used to expand the document selection. Related/broad queries are typically used with organic document sets, not paid advertising results.
So, while not spelled out definitively, the patent language strongly suggests that the query-responsive documents selected to generate summaries are organic search results rather than paid or promotional results.
Friends PDF
Friends PDF is a desktop application that offers 30 free questions to try. It requires a ChatGPT API key for these tests.
Once installed, PDF Pals can process uploaded PDF documents.
The tool helpfully includes page numbers with its answers, making it easier to verify the information. Clicking on any page number will take you to that section of the document.
In my testing, PDF Pals did not simplify the patent to my level of understanding. His answers were too technical, despite my insistence to the contrary. The summaries were nevertheless useful, if complicated.
Ask your PDF
Ask your PDF is a web application that does not require any API keys for testing. After you scan a document, AskYourPDF suggests optional follow-up questions. Like PDF Pals, it includes page numbers, although they are not clickable.
AskYourPDF’s answers were easier to understand than those of PDF Pals and, conversely, less complete. And it did not extract definitions from the patent PDF, incorrectly stating that there were none.
So AskYourPDF in my testing was useful for higher level overviews, but not detailed. One benefit of this approach is likely to be a reduction in hallucinations.
Interestingly, all four tools analyzed the Google PDF patent slightly differently. Each provided unique explanations. The key is to verify the information. All tools made errors.