Building a Local RAG: An AI Engineering Reality Check
I just finished experimenting with a RAG-based chat system powered by llama.cpp and the Hermes-3-Llama-3.1-8B-GGUF (Q4_K_M) model.
Spinning up the local inference server required a few specific flags to ensure compatibility with the langchaingo orchestrator—specifically enabling embeddings and configuring mean pooling:
llama-server -hf NousResearch/Hermes-3-Llama-3.1-8B-GGUF:Q4_K_M --port 8080 -c 8192 --embeddings --pooling mean
It was quite a journey. I started with basic similarity searches against a pgvector database and eventually had to implement a ReAct loop to make the chatbot respond more intelligently. The difference in output quality was significant.
The system performs surprisingly well when running locally. However, the thought of deploying this as a multi-user production system is a completely different story. The operational costs and infrastructure headaches to scale LLM inference are massive.
Regardless of the deployment challenges, building this small RAG system from scratch was incredibly valuable. It gave me a much more grounded, realistic perspective on the day-to-day realities of being an AI engineer.