Since my last blog post in June, the Infrastructure team has been hard at work and now we finally get to take the wrapping off one of the projects we completed earlier this year.
It is my pleasure to announce iovation's third primary data storage datacenter colocated with Opus Interactive. This is iovation's 5th global datacenter and is used to store a third copy of key data elements separate from our other two primary data storage datacenters by using “Replication Factor 3”.
What is Replication Factor 3?
Replication Factor 3 is a term that has meaning when talking about the Apache Cassandra datastore. In Cassandra, data written into the datastore is divided up and stored on different nodes (servers) of the cluster. For resiliency purposes you can choose how many different nodes data is placed on. By having a Replication Factor greater than 1, we are able to withstand the loss of a single server without losing data or causing the system to become unavailable.
As an example, in the diagram below the blue circles represent nodes and the quarters of the large circle each indicate one fourth of the records stored in the system. You can see that the quarter of the data represented by “A” is stored on the top, right, and bottom nodes in the “ring”.
How is this Deployed?
In addition to handling the failure scenario of a single node going down, or a given server "rack" going down (due to a power or networking issue), iovation takes this one step further and places our three nodes in geographically separate locations. This allows us to handle the loss of a datacenter and still retain server level redundancy. The converse is also true, we can handle the loss of a server without losing our datacenter level redundancy.
Why Replication Factor 3?
When you have hundreds of servers (as iovation does), at any given time you are guaranteed to have at least one node (server) offline for maintenance. This extreme level of redundancy allows our operations team to sleep well at night without concern that we will fail to serve our customers due to unexpected faults.
Opus Datacenter Overview
iovation’s new datacenter is located in a building known as Brew Block 1 (BB1) which is part of the former Henry Weinhard Brewery complex. It was completely rebuilt in 2002 as a telecommunications carrier hotel with no expense spared.
Above you can see some of the primary electrical switchgear including two of the four 2.5MW utility step down transformers.
Here you can see one of the banks of inert gas that works as part of their fire suppression system.
And here is one of the massive chiller units that not only services the datacenter, but also the four other Brewery Block buildings. There is ample amounts of excess cooling capacity (4000 tons total) as the chiller plant was built to serve as the district cooling plant for this section of Portland.
While this project has been a fun one and the infrastructure team is very proud of its success, it is just one piece of some much larger projects we have been working on all year. I look forward to a future blog post with more exciting news in the coming months.