Jina AI + Elastic → Efficient Data Storage in the Cloud

Introduction

DocArray is a data structure that can accommodate various data types, including text, images, audio, and video. It is part of the Jina AI ecosystem and is designed to be easy to use with Python.

Elasticsearch is an opensource ecosystem of specialized tools and extensible APIs that make it easy to build search solutions. It is completely configurable to build NLP driven vector search for an enhanced user experience. It is a distributed document store. Instead of storing information as rows of columnar data, Elasticsearch stores complex data structures that have been serialized as JSON documents making it highly scalable and fault-tolerant.

Elasticsearch backend provides the much-needed scalability for Jina search applications and can be used as a document store for DocumentArray. It facilitates faster retrieval of documents on search tasks that include embeddings.

Set Up the Elastic Service

We are going to use Elasticsearch as a storage backend and to do this first the Elasticsearch service needs to be started. You can do that by creating the docker-compose.yml as follows:

Once you have created the docker file, you can run docker compose up to start the instance.

Minimum Working Example

In this example, we will create a weaviate local instance to store the Document and build a simple text search.

First, start the Elasticsearch service and create a DocumentArray array instance.

Now let’s index the Documents:

Now we will generate the embeddings using the BERT model:

Finally, we can query the indexed Documents and get the results:

Output: Persist Documents with Elasticsearch.

Learning Resources

This is the final chapter of three-part series about exploring different cloud document stores that can be used with Jina’s DocArray.

Venture into the exciting world of Neural Search with Jina’s Learning Bootcamp. Get certified and be a part of Jina’s Hall of Fame! 🏆

Stay tuned for more exciting updates on the upcoming products and features from Jina AI! 👋

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Shubham Saboo

Shubham Saboo

239 Followers

🥑 Head of DevRel at Tenstorrent | 📕 Author | ✍️ Simplifying ML and NLP one blog at a time!