Jina AI + Elastic → Efficient Data Storage in the Cloud
Get faster retrieval on embeddings with Elasticsearch as the document store for Jina’s DocumentArray!
Introduction
DocArray is a data structure that can accommodate various data types, including text, images, audio, and video. It is part of the Jina AI ecosystem and is designed to be easy to use with Python.
Elasticsearch is an opensource ecosystem of specialized tools and extensible APIs that make it easy to build search solutions. It is completely configurable to build NLP driven vector search for an enhanced user experience. It is a distributed document store. Instead of storing information as rows of columnar data, Elasticsearch stores complex data structures that have been serialized as JSON documents making it highly scalable and fault-tolerant.
Elasticsearch backend provides the much-needed scalability for Jina search applications and can be used as a document store for DocumentArray. It facilitates faster retrieval of documents on search tasks that include embeddings.
Set Up the Elastic Service
We are going to use Elasticsearch as a storage backend and to do this first the Elasticsearch service needs to be started. You can do that by creating the docker-compose.yml as follows:
Once you have created the docker file, you can run docker compose up to start the instance.
Minimum Working Example
In this example, we will create a weaviate local instance to store the Document and build a simple text search.
First, start the Elasticsearch service and create a DocumentArray array instance.
Now let’s index the Documents:
Now we will generate the embeddings using the BERT model:
Finally, we can query the indexed Documents and get the results:
Output: Persist Documents with Elasticsearch.
Learning Resources
This is the final chapter of three-part series about exploring different cloud document stores that can be used with Jina’s DocArray.
Venture into the exciting world of Neural Search with Jina’s Learning Bootcamp. Get certified and be a part of Jina’s Hall of Fame! 🏆
Stay tuned for more exciting updates on the upcoming products and features from Jina AI! 👋
