How Teracloud helped

migrate to the Cloud

SocialRank is a data analytics company with the easiest way to identify, organize, and manage your followers on X (Twitter) and Instagram. Their core business is based on a medium/large-sized ElasticSearch cluster holding 22TB of data and growing. Teracloud managed to lower costs and improve the security and reliability of the platform by migrating it to the AWS cloud.

About Social Rank

SocialRank allows you to:

Identify: It provides the opportunity for consumer opt-in: allowing you to generate ethical, high-quality first-party data. With the power to export and re-market to valuable target audiences.
Segment: Discover new audiences, segment, and filter across industries and interests with bio keywords, geographic breakdown, hashtag usage, activity rate, post times, and more. Understand audience overlap across a portfolio of brands — yours, competitors, and more.
Engage: Activate an audience whenever and however. Drive calls to action like new platform follows, newsletter sign-ups, e-commerce, and more. Engage with ease to re-market and prompt impactful results.

Social Rank's challenges

SocialRank infrastructure was completely deployed on-premise, with all the disadvantages of this situation. There were multiple custom-made data ingestion processes connecting to Twitter and Instagram backbones to feed the large ElasticSearch cluster and serve the customers. SocialRank contacted us because the base Kubernetes cluster lost its only Master node and they lost control of their cluster, it was just a matter of time for the whole infrastructure to go down. The challenges can be summarized as:

Security:
- Deprecated Kubernetes version: All Kubernetes manifests were written for Kubernetes 1.4
- Deprecated Elasticache version: System running on Elasticache v5

Reliability:
- Incomplete backup strategy
- No DR strategy defined. No resources devoted to DR
- No node replacement strategy. Hardware failures introduced an extremely high risk of platform performance degradation.
Performance:
- Scalability: Very high costs for scalability because of on-premise hardware.
Operational:
- Underlying tools versioning results in increasingly high difficulties to upgrade software and apply patches. Also, no modern tools (like SecretsManager) can be used because of the same reasons

Why AWS

AWS was chosen because it helps create a secure, high-performance, resilient, and efficient Kubernetes cluster, which is the base of the whole infrastructure. Using EKS, AWS Lambda, Route53, ACM, S3, and other AWS-managed services allowed us to plan a smooth and safe migration, deploy the infrastructure in parallel to the prior on-premise workloads, test it fully, and earn the trust of the engineering and management teams to flip the switch.

Why Teracloud

SocialRank needed a trusted AWS partner with a strong background in cloud migrations. Even more importantly, given this was a business continuity risk scenario, they couldn’t afford mistakes, delays, or contingencies that may result in SocialRank's customers losing access to the platform and losing data. Fortunately, our Cloud engineers have proven their capabilities to SocialRank in past projects along with high implementation speed in similar scenarios with other clients.

Our solution

The solution was structured around 6 core aspects:

Core customer requirements
Security
Performance
Reliability
Cost optimization
Operational excellence

We used the 6 pillars of the Well-Architected Framework along with the core customer requirements to design the solution. Given that the risk increased with time (any pod malfunctioning on the on-premise infrastructure won’t be recreated because the master was down), we designed the solution based on three principles:

Speed: Choose the solution that fulfills the requirement faster
Reliability: Customer data is the maximum asset, any solution that introduced risk here was discarded
Security: Customer data should be protected at rest and in transit, with no exceptions.

In this design, we leverage the power of AWS services to reach our goals faster and with fewer costs:

The core element is the EKS cluster because all services are run within it. The backend, frontend, and ElasticSearch databases run inside the cluster. The AWS ElasticSearch service could not be used in this case because the client required a custom ES plugin, which is currently not allowed on AWS ElasticSearch.
The ElasticSearch pods require storage space, which is provided by EBS volumes attached to the EKS nodes.
All components except AWS Application Load Balancer were located on private networks, and ssh access was provided by AWS SystemsManager.
All secrets were migrated to AWS Secrets Manager, resulting in simpler architecture for the Kubernetes internals.
Route53 provides DNS resolution.
AuroraRDS for mySQL was the preferred approach to migrate a support mySQL database from on-premise.
The deployment of new app versions is handled with a mix of CodeBuild and Flux for Kubernetes, with the aim of getting a completely automated CI/CD environment.

Benefits for Social Rank

Once the switch was flipped, the immediate benefit was non-tangible: the business continuity risk was reduced to near zero. Additional benefits are listed below, grouped by area of impact:

Reliability. Using this new architecture design and implementation we achieved the following benefits:
- 99.999% availability of the infrastructure
- Scaling capabilities. The current application implementation does not allow for automatic scaling, but the work for coordinating the scaling of infrastructure and application has been greatly reduced.
- EKS provides 3 master nodes, eliminating the risk that created this project.
- Automatic snapshotting and backups of EBS volumes (for ElasticSearch data) and the RDS instance (for SQL data).
Performance. Using smaller instance sizes (comparing CPU and memory with on-premise) the performance of the system was improved.
Security. Teracloud has implemented the best practices when designing the infrastructure and the data pipelines:
- Using IAM fine-grained permissions for roles.
- Using a 2-tier architecture for the VPC, and isolating the data in the more restricted data zone.
- Enable a data-access policy for S3 buckets access, including backups.

Moving forward

To complete the migration on time some compromise decisions were made, which resulted in a less-than-ideal scenario. While we did not introduce security or reliability risks, some areas can certainly be improved:

Costs: The current instances can be migrated to Graviton, which will reduce costs. This initiative is currently blocked until we upgrade to ElasticSearch v8.
Performance: Application refactoring can result in better usage of cloud resources. Also, it can allow us to auto-scale.
Operational: Having an automated CI/CD pipeline will allow us to be more confident in the process of deploying new applications and versions.

Our approach

Teracloud is a fast-growing AWS Advanced Tier Services Partner created by certified cloud experts, with seasoned experience in migrating and deploying startups, enterprises, and everything in between to the cloud. We've worked with companies from many industries such as airlines, healthcare, education, and e-commerce designing, implementing, and managing Cloud workloads with HA architecture under 99.9% uptime SLA and PCI/HIPAA compliance requirements.

Our expertise ranges from designing state-of-the-art cloud architecture to deploying and managing cloud environments through Managed Services including AWS Consulting, Architects, DevOps, DevSecOps, MLOps, 24/7 Support, Automation, Disaster Recovery, and Security. We provide secure, scalable, and highly available solutions for our clients.

Get in touch

Take your business to the next level.