Key Aspects of Building Advanced Search-Assisted Chatbots

Introduction to Generative Chatbots

Chatbots are gaining popularity as productivity tools in companies. They help in finding information from various fields, such as HR, IT, sales or engineering. With the introduction of Chat-GPT technology and vector databases, it has become possible to create chatbots that generate accurate and consistent answers by combining LLM and RAG models.

Example of use case at NVIDIA

NVIDIA has built three chatbots: NVInfo Bot, NVHelp Bot, and Scout Bot, using the NVBot platform. Each of these chatbots supports different areas of information: from general corporate data, to IT and HR support, to financial information. Each bot uses advanced technologies and processes that allow it to operate effectively in different contexts.  

Ensuring the current status of company data (F)

A key challenge is ensuring that LLM models have access to the latest data. Base models are often static and can generate incorrect answers. The RAG process involves taking up-to-date information from the vector databases and feeding it to the LLM to generate answers. NVIDIA uses various techniques, such as metadata enrichment, document fragmentation, query paraphrasing, and result re-ranking, to improve the accuracy of the answers.

Building Flexible Chatbot Architectures (A)

In a dynamic technological environment, it is crucial to create flexible platforms that can adapt to new tools and technologies. NVBot is a modular platform that allows for the selection of different LLM models, vector bases and agents, while supporting security, authorization and monitoring. This allows for the creation of chatbots that can evolve with technological advances.

Economics of chatbot implementation costs (C)

The cost of implementing LLM-based generative chatbots can be high. NVIDIA considered both large, commercial LLM models and smaller, open-source models that offer comparable quality at a lower cost. A key aspect is also implementing an internal LLM gateway that helps manage subscriptions and data security.

Testing RAG-based chatbots (T)

Testing generative AI solutions requires long test cycles and human validation of responses. Test automation and the creation of representative reference data sets are key to effective testing. NVIDIA implements feedback mechanisms and automated testing to continuously improve chatbot performance.

Securing RAG-based chatbots (S)

Trust in generative chatbots is key, so it’s important to implement safeguards against hallucinations, toxicity, lack of fairness, lack of transparency, and security breaches. NVIDIA focuses on securing access to corporate content and implementing compliance policies to ensure safe use of chatbots.  

The future of chatbots

In the future, RAG-based chatbots will become increasingly sophisticated and complex. Work will focus on developing agent architectures that can handle complex, multi-part queries and provide analytical responses. It will also be important to efficiently summarize large amounts of frequently updated enterprise data and automate various RAG checkpoints to optimize the entire process. As the technology evolves, chatbots will play an increasingly important role in increasing the efficiency and accessibility of information in the workplace.

Summary

NVIDIA presents the FACTS framework, which covers the most important aspects of building chatbots: data freshness (F), architecture (A), costs (C), testing (T), and security (S). NVIDIA shares its experiences and strategies for optimizing chatbot performance at each stage. Future work will focus on developing agent architectures to handle complex queries and automate various RAG checkpoints. Creating effective RAG-based chatbots requires careful planning, continuous evaluation, and optimization. It is also crucial to take into account specific security requirements and cost management. Source: "FACTS About Building Retrieval Augmented Generation-based Chatbots„