Overview
Writing test case based on provided requirements can be a tedious and time-consuming process, especially when the design documents are only partially written. To alleviate this, an automated soultion would be desired. This solution needs to be robust enough to fill requirements from missing design documents whilst adhering to strict design rules. This is why we decided to develop a custom Retrieval-Augmented Generation (RAG) system powered by a Large Language Model (LLM) with the ability to parse client-provided design documents.
Goals
The goal of the project is to create an automated solution that enables quicker test case design by providing requirement-specific action and verification steps. The soulution should enable quicker test case development and assist with generating needed documentation. It should be apable of generating necessary test cases even from partial documentation.
Solution
Our solution involves creating an LLM that uses document embeddings to leverage knowledge not seen during fine-tuning (new design documents) and generate meaningful responses from the retrieved context. The system utilizes Milvus for vector database embeddings of the documents, LangChain for RAG system integration with chat history, and Ollama as the LLM inference server. The chosen LLM is the open source Mistral LLM, fine-tuned on customer-specific design documents and format output. A simple UI was integrated, allowing new documents to be uploaded and prepared for LLM use.
Results
- LLM successfully answers design-related prompts
- Responds to previous interactions effectively
- Properly utilizes retrieved documents for generating test cases
- Enables quicker test case development
- Facilitates the generation of needed documentation even from partial design documents
