Good chunks make good RAGs.
Chunking, embedding, and indexing are essential facets of RAGs. A RAG app that makes use of the suitable chunking approach performs nicely when it comes to output high quality and velocity.
When engineering an LLM pipeline, we use completely different methods to separate the textual content. Recursive character splitting is the preferred approach. It makes use of a sliding window method with a hard and fast token size. Nonetheless, this method doesn’t assure that it may well sufficiently maintain a theme inside its window dimension. Additionally, there’s a danger that a part of the context falls into completely different chunks.
The opposite approach I like is semantic splitting. Semantic splitting breaks the textual content each time there’s a major change between two consecutive sentences. It has no size constraints. So, it may well have many sentences or only a few. Nevertheless it’s extra prone to seize the completely different themes extra precisely.
Even the semantic splitting method has an issue.
What if sentences removed from one another are nearer of their that means?