top of page
Writer's pictureVirginie Wylleman

Chatting with your documentation: Today’s capabilities explored




Making documentation come alive

With AI advancements coming our way at a rapid pace, we find ourselves both compelled and excited to explore the opportunities they present for us and our customers. While AI can already assist in automating tasks and enhancing the writing processes, imagine the potential if we used AI to optimize the very products we offer, i.e. documentation. Could we use AI to transform web-based documentation from a static resource into a dynamic, interactive conversation with the end user? Is this feasible, and more importantly, would it truly add value for our customers? Let’s find out.


Selecting tools for analysis

Given our existing licenses and current website chatbot, we decided to select Copilot Studio and CustomGPT for further exploration. Both tools offer LLM-based solutions for handling user questions about domain-specific content:

  • CustomGPT is a smart AI-powered Q&A bot which helps you quickly consult documentation and generate responses in a customizable tone and format. It’s an ideal choice for businesses aiming to improve their customer support with dynamic, context-aware interactions.

  • Microsoft Copilot Studio is an AI tool that provides a Q&A bot while allowing you to log questions and feedback. This helps identify pain points and missing information, allowing you to optimize both the bot and the documentation. With customizable flows, it can handle unique inquiries, provided you have a clear understanding of user needs. Additionally, connectors and actions make copilot studio more than just a chatbot, expanding its functionality beyond simple retrieval-augmented generation (RAG).


Riding a bumpy road

Unfortunately, the road to conversational documentation proved more challenging than anticipated. Despite its promising potential, the technology still lacks some functionality and faces minor bugs that currently make it too early for us to offer our documentation in this format.


Copilot Studio, for instance, quickly fell short due to its lack of accuracy and consistency, especially when compared to CustomGPT. It struggles to integrate and search across different sources of information. For example, we couldn't teach it terminology to apply during its search for relevant content in the documentation.


CustomGPT, on the other hand, handles both areas remarkably better. Copilot studio would sometimes claim not to know the answer to a question despite having the knowledge. On re-prompting, it might then provide the correct answer, leading to inconsistent response quality. Additionally, some topics were inappropriately triggered, and this issue couldn’t be fixed after reaching out to Microsoft.


However, CustomGPT isn’t without flaws. It can only cite entire documents, not specific chunks of information. Unlike Copilot Studio, CustomGPT has fewer options for detailed insight collection. It doesn’t inherently have connectors that can perform actions such as structuring and storing user questions and feedback.


Finding the silver lining

We have to give credit to CustomGPT for delivering high-quality answers. After using the chatbot for over a year, we’ve noticed the tool improve significantly. Its ability to integrate data from multiple sources is especially remarkable. This feature allows the chatbot to pull relevant information from various places, resulting in a deeper understanding and more accurate responses.


On the flip side, Copilot Studio shines with its options for guiding conversations and offers more straightforward insight collection opportunities.


Final thoughts

If your knowledge base is in SharePoint and you need a tool for handling simple user queries or automating tasks based on user intent, Microsoft Copilot Studio is a solid choice. It excels at answering specific questions, such as HR policies or helping customers book trips at a travel agency. However, if you need a tool to extract relevant information from large documents, consider one that specializes in retrieval-augmented generation (RAG), like CustomGPT.


While neither tool is perfect for our objective, CustomGPT has improved significantly in recent months. Copilot Studio, with its different focus, may take longer to meet our needs.

 

In summary, Copilot Studio is better for internal efficiency, while CustomGPT is more suited for creating content for external publication. Although Copilot Studio can be used for external work, it isn’t designed to function solely as a Q&A bot.


bottom of page