Why we can consider migrating from the Algolia search service to Elasticsearch?
What you need to know before and how implemented it.
Full-Text services are really important?
First o all, full-text search on websites or applications is one of the fundamental features you need to consider when deciding to have an online presence.
In this field, Algolia is the best service available in a very competitive market.
So why were we asked to migrate from Algolia to Elasticsearch?
Let’s start by considering the functionalities that Algolia provides.
Algolia functionalities
Talking about web applications, the functionalities that Algolia makes available to us are unique, mainly due to the client libraries that Algolia itself releases for the integration in the various platforms.
These libraries allow real-time searching databases of millions of records, making the document search experience unparalleled in accuracy and speed.
Algolia also provides numerous features available directly in the control panel.
Among them:
- Instant search
- Synonym suggestions
- Visual editor
- Custom rules
- Personalizations
- Algolia AI
- AI search
All these features lead to amazing results, as you can read on the ALGOLIA website:
- 5X faster development
- +30% revenue
- +10% page views and subs
From all this, it is clear that the advantages of implementing Algolia are undoubtedly significant.
To understand if Algolia is suitable for us, we must also consider another aspect that is not secondary.
Like all SaaS, the cost of Algolia’s service is calculated based on usage; in particular, above 10,000 searches per month, fees are charged for packages of 1,000 searches.
It’s easy to understand that this cost is sustainable for e-commerce, especially considering the increase in turnover that Algolia brings.
In the case of a service for which a monthly subscription is paid, it becomes a cost item to be considered very well.
It is, in fact, very easy to reach 1,000€ and more of cost per month.
Migrating to an alternative search service: Elasticsearch
In our case, the client needed to reduce search costs to prioritize other types of development.
They asked us to move their search system to an alternative service like Elasticsearch.
A further indication was to test the search on the MySQL database of their web application anyway, a Laravel-based installation with some years on its shoulders.
Issues to be resolved in migrating the Algolia database to the Elasticsearch database
After a careful analysis of the data contained in the Algolia database, about 1.5 million records, we identified the main critical issues:
- The creation of a script that would export records from Algolia and import them into Elasticsearch.
- The realization in Laravel of a query service on Elasticsearch that would reproduce as much as possible the characteristics of Algolia, both in terms of speed and intelligence in the full-text search.
- The replacement of the data loading functionalities on Algolia with Elasticsearch functionalities.
Migration from Algolia to MySql
As requested by the customer, we tested the possibility of using a MySQL database.
One very positive thing about Algolia is that the data export process is very clear and efficient.
Through an API call ‘browse,’ the whole database is exported at an amazing speed.
Unfortunately, we cannot say the same for MySQL.
Once the table structure was reproduced (through full-text indexing, inno-DB), the data extraction script from Algolia to MySql was stuck around 300.000 records, even though it was running on a Digital Ocean instance with 2 processors and 8Gb of memory with Ubuntu 20.04.
We had, therefore, to create a script that could load only 100,000 records at a time.
In this way, we were able to complete the entire data transfer in about 7 hours.
However, when we started to test MySQL full-text queries (match… against) on a table of about 10 Gb, several problems occurred:
- Very slow execution.
- Inaccurate execution.
- Increase in duration as the number of words searched increases.
- Other minor inconveniences of various kinds.
Having verified the inefficiency of MySQL for this type of activity, we proposed to the customer the adoption of Elasticsearch.
Migration from Algolia to Elasticsearch
We then proceeded to install Elasticsearch using Docker.
As a cloud service, we used EC2 Small Amazon AWS.
To perform the migration, we took advantage of the bulk APIs made available by Elasticsearch, and it was speedy.
In testing, we found that the search on Lucene, the search engine used in Elasticsearch, is very accurate and fast.
The only difference is the lack of real-time search suggestions that you can implement with Algolia.
Overall, we can say that Elasticsearch can easily compete with Algolia at a competitive price in terms of pure performance.
