Quickwit takes on Elasticsearch, an open source search engine for large datasets

Meet today’s leading leaders online at the Data Summit live now! Look here.


leave him OSS Enterprise Newsletter join your open source journey! Sign up here.

Search plays a fundamental role in almost every modern application, from Amazon and Netflix to Slack and Salesforce. In addition, each application generates swathes of log data that contain time-stamped information about events within the software — these can be details about what resources have been accessed, an application’s runtime characteristics, and anything pertinent to the operation of that system.

The ability to search through and understand all of these computer-generated logs is important as it helps organizations troubleshoot and troubleshoot, resolve bottlenecks and latency issues, comply with regulations or internal security policies, and better understand what’s going on under the hood goes. All of this is part of what is known as “observability” – ie the ability to measure the internal state of a software system by analyzing the raw outputs.

Typically, companies pool all of their log data in a centralized database system like ClickHouse. However, managing and managing all of this comes with many challenges, while the costs associated with storing log data can cause organizations to drop parts of it. This is a problem Quickwit aims to solve with an open-source, cloud-native search and analytics engine designed for large datasets.

Founded in 2020, Quickwit touts its ability to run subsecond queries on terabytes of data on object storage services like Amazon S3 — and promises to do it up to “ten times cheaper” than Elasticsearch. But Quickwit isn’t necessarily designed to be a direct replacement for every scenario covered by mainstream search providers like Elasticsearch – it has more limited, more targeted use cases in mind.

To build on the momentum it has built since its initial release last July, the company today announced a $2.6 million seed funding round, co-led by FirstMark and firstminute, which will include a number of Notable Angel contributors, including MongoDB co-founder Eliot Horowitz; Dataiku CEO and CTO Florian Douetteau and Clément Stenac; and SendGrid founder Isaac Saldana.

‘Cost Efficient’

Paul Masurel: Co-founder and CEO of Quickwit and creator of Tantivy
Paul Masurel: Co-founder and CEO of Quickwit and creator of Tantivy

Built on the open source, Rust-based Tantivy search engine library created by Quickwit CEO Paul Masurel five years ago, Quickwit is a distributed search engine that offers additional functionality on top of Tantivy’s “low-level building blocks”. for the search. Quickwit provides a REST API for indexing data, performing search queries, managing indexes, and managing clusters with a set of pre-built connectors that span data sources such as Apache Kafka, Amazon Kinesis, and Amazon S3.

At its core, Quickwit targets so-called “immutable” records (data that is never deleted or updated), making it perfect for organizations that want to store and search log data. It also promises sub-second latency of just 140 milliseconds, which is fine for log management searches. However, it is well known that milliseconds matter in the online world, which is why Quickwit does not target use cases like e-commerce websites that require lower latency.

“A search query requires at least two round trips to object storage, but object storage systems have higher latencies than local disks used by systems like Elasticsearch,” QuickWit co-founder François Massot told VentureBeat. “The bottom line is that Quickwit can never respond faster than 130-140ms, which is acceptable for the use cases we’re targeting, but not for e-commerce applications where higher latencies correlate with lost revenue.”

Quickwit co-founder Francois Massot
Quickwit co-founder Francois Massot

All in all, Quickwit competes with companies like Elasticsearch for a few purposes, including searching logs, cloud storage backups, and providing full-text search capabilities for online analytical processing (OLAP) databases like ClickHouse. But it’s in these use cases that Quickwit hopes to differentiate itself enough to win the hearts and minds of small businesses and corporations alike.

First of all, object storage is cheaper than disks to store data, while Quickwit is written in Rust, which is known to consume less memory than Java, which Elasticsearch is based on. Additionally, Quickwit’s decision to separate processing power and storage may set the company apart in the analytics space — it claims to be the first open-source search engine with such an architecture.

“Quickwit instances are stateless and can be started or shut down in seconds – you don’t have to move data like in Elasticsearch because storage is separate from computation,” explained Adrien Guillo, co-founder of Quickwit.

blank
Quickwit co-founder Adrien Guillo

In theory, this all means faster and cheaper for Quickwit’s target use cases. And given this cost-efficiency promise, organizations may be more inclined to retain more log data, which will improve the insights they gain into their system’s performance.

“Many organizations end up reducing their log retention to contain their costs,” added Massot. “Quickwit eliminates the need to throw away this valuable data.”

In terms of the types of businesses Quickwit targets, Guillo argues that it will suit businesses of all sizes. Smaller companies will want to use Quickwit as a building block for their log search observability, while larger companies will build entire applications on top of Quickwit that include application and log management, search analytics, data lake search and analytics, and more.

“Enterprises struggle to operate existing search systems at scale and must mobilize significant resources and capital to do so, particularly for the ever-growing number of logs generated by applications, systems and business events,” said Guillo. “Quickwit offers unprecedented cost efficiency.”

While Quickwit didn’t reveal much in terms of its early customer base, Guillo confirmed that they’ve been working with French unicorn Contentsquare on a proof-of-concept.

Right now, Quickwit’s business model is based on a simple two-license approach – an open-source AGPL license for free use and a commercial license that includes support and gives the licensee a “voice on our roadmap,” Guillo said.

While the company may consider offering SaaS in the future, that’s not currently on Quickwit’s agenda.

VentureBeat’s mission is intended to be a digital marketplace for technical decision makers to acquire knowledge about transformative enterprise technology and to conduct transactions. Learn more

https://venturebeat.com/2022/03/10/quickwit-takes-on-elasticsearch-with-an-open-source-search-engine-for-large-datasets/ Quickwit takes on Elasticsearch, an open source search engine for large datasets

Chris Barrese

InternetCloning is an automatic aggregator of the all world’s media. In each content, the hyperlink to the primary source is specified. All trademarks belong to their rightful owners, all materials to their authors. If you are the owner of the content and do not want us to publish your materials, please contact us by email – admin@internetcloning.com. The content will be deleted within 24 hours.

Related Articles

Back to top button