Lately, we’ve gotten some notice for upgrading to an open-source stack, including Apache Cassandra, which we chose for its linear scaling capability, speed, and fault tolerance.

Another open source tool that we’re using increasingly at iovation is Elasticsearch. It’s a highly scalable, distributed, search and analytics engine based on Lucene with a RESTful web interface. Elasticsearch is written in Java and has a very active development community.

Our first use of Elasticsearch started over two years ago in combination with Logstash to organize access to our growing number of server and application log files. Recently we’ve added Kibana 3 as the UI to complete our “ELK” stack, bringing it all together visually.

We presently have:

  • 2.8 terabytes of log information on two independent Elasticsearch single node clusters
  • 1.1 billion total log events for a rolling 14 days
  • 77 million events a day coming in from over a thousand of our servers and other devices

Some of the ways we use ELK:

  • Monitoring health of production systems and load balancers
  • QA testing and analysis
  • Regression analysis
  • Researching log information for our clients

We’ve recently implemented Elasticsearch to handle the transaction history search in our SaaS fraud prevention admin console aka “the admin” which is used by clients to access our services. Elasticsearch was chosen here for its scalability, fault tolerance, and native searchability. Our clients need the ability to search on any field in the document. Also, the time series nature of transaction history is very similar to that of our log data, making Elasticsearch a natural fit.

Our Elasticsearch setup for the admin consists of two independent 12 node clusters, with one datacenter located in Portland and the other in Seattle. Client requests are load balanced between the two clusters. We can lose either cluster, plus a server node in the remaining cluster, and still be able to be able to handle requests against a full copy of the data.

Note that the consistent configuration of the many servers required by our scalable architecture would be a difficult task if not for our use of another open source product, Puppet, but then that’s a whole topic in itself.

Since switching to this Elasticsearch configuration there has been a significant performance boost for our client transaction searching.

The admin’s rolling six months of client events currently comprises:

  • 1.2 terabytes of event data
  • 1.7 billion event documents

We’ll soon be deploying an Elasticsearch solution to our admin's account and device velocity reports. This will take advantage of the aggregation feature that automatically ranks results, returning the highest associated accounts and devices.This will add scalability and enhance the flexibility and timeliness of these reports.

We are currently researching use of the Elasticsearch significant terms feature to discover the “uncommonly common” data distributions, with the hope of helping our clients uncover more potential fraud. We also plan to explore applications for the similarity module, which enables scoring the “likeness” of events.

We’re just beginning to scratch the surface of what this powerful search tool can really do. We will continue to look at the different ways we can apply search to give our end-users more information faster.

Lastly, we know we’re in good company using Elasticsearch when we see other organizations with massive amounts of data to analyze, index and search like Wikipedia, Netflix and Facebook, choose the same open source tool we’re using here at iovation.