We leveraged just two dual-core VMs and a Nodebalancer to process a pretty impressive number(40K) of movie reviews in just 20 minutes. However, deploying in a single region can lead to latency/availability issues and doesn’t fully utilize the global reach of modern cloud infrastructure Akamai Connected Cloud offers.

Also, single region deployments are kinda boring

In this post, we’ll expand our deployment by setting up virtual machines in all available Linode regions and replacing the current Nodebalancer with Akamai’s new Cloud Load Balancer (currently in beta access).

What is Akamai’s new Cloud Load Balancer you may ask? It’s really cool peace of tech.

Think of it like an umbrella over the internet; it gives you the possibility to load balance your workloads across ANY location; it can be on prem, Akamai’s cloud, some other hyper-scaler, heck, it can even be your home IP address if you want to

As long as it can be reached over the internet, Cloud Load Balancer can use it to deliver the request.

Joking aside, here’s a more official description of the service:

The Akamai Cloud Load Balancer (ACLB) (formerly referred to as Akamai Global Load Balancer) is a layer 4 and 7 load balancer that distributes traffic based on performance, weight, and content (HTTP headers, query strings, etc.). The ACLB is multi-region, independent of Akamai Delivery, and built for East-West and North-South traffic. Key features include multi-region/multi-cloud load balancing and method selection.

Why Scale Globally?

Scaling out our application across multiple regions has several benefits:

1. Reduced Latency: By having servers closer to our users, we can significantly reduce the time it takes for requests to travel back and forth.
2. High Availability: Distributing the load across multiple regions ensures that if one region goes down, well, we kinda don’t care, our app stays online
3. Better Performance: I mean, you can’t beat physics; simply having the possibility to do compute closer to the user improves performance and user experience.

Step-by-Step Deployment Guide

Ok, let’s get into the meaty part; codebase from our previous post hasn’t changed significantly; only thing which we changed is that we’re not hardcoding out region anymore, but we are fetching the list of available regions from Linode API and deploying an instance in each region.

In the code which fetches the regions, you will notice that I commented out the authentication part.

Some regions are available only to authenticated users; if you’re one of those, just uncomment those few lines and the full region list will be returned to you.

Let’s start with terraform plan and see what we will create.

terraform plan

Ok, 25 instances, just what we expect since Akamai has 25 compute regions currently publicly available.

Let’s proceed with terraform apply

terraform apply

I don’t know about you, but I always nerd out on seeing a bunch of servers popping up in the console

In a minute or two we should have all instances deployed. After the instances have been deployed, cloud-init will kick off and install DeepSparse server with an Nginx proxy in front (check out the previous post or the YAML file in the repo for more details).

Awesome! After we’ve got the infrastructure up and running, last step is to add our nodes to the Cloud load balancer pool; at the moment we will need to do some Clickops CLB service is currently in beta so IaC support isn’t out yet.

First step is creating a Cloud Load Balancer by clicking the “Create Cloud Load Balancer” button and giving it a name.

During the beta period, Cloud Load Balancer is deployed only in 5 locations. This number will grow drastically once the service goes GA.

With Akamai’s Cloud Load Balancer, everything starts with a “Configuration“. Let’s create one by pressing “Add Configuration” button.

We will configure our load balancer to “listen” on both HTTP and HTTPS. Once we selected HTTPS as our protocol, we need to add a certificate.

In order to do that, we need to prepare our certificate and private key which we will paste into the configuration field. In this case I will use a self-signed certificate.

At this stage we will only cover the configuration for the HTTPS protocol, HTTP is really easy and won’t bother wasting your time on it.

We need to paste in the certificate & the key, enter the SNI hostname and press “Create and Add” button.

After we’ve got the configuration and the certificate added, we need to create a “Route“. Let’s click on “Create a New HTTP Route” button and give it a name.

Great, we’ve created a route, but the route is currently empty and it doesn’t route anything. We will come back to this a bit later.

Next step is to save our configuration and click on “Service Targets” tab.

This is the place where we will define our target groups and origin servers. Click on “New Service Target” button

Next steps are quite explanatory, we need to give it a name and add the nodes which we want to load balance across.

Remember, this can be one of the existing Linode instance or it can be ANY IP address which can be reached via the internet.

This is the step where I could really use IaC support, we need to add all 25 servers by using ClickOps to our “Endpoints” list.

This is also the place where you can also select the load balancing algorithm which will be used to balance requests between the nodes. At the moment there are 5 of them available:

• Round Robin
• Least Request
• Ring Hash
• Random
• Maglev

Last step in “Service Target” configuration is to set the path and host header we will use for the health checks on the nodes and click on “Save Service Target” button.

We’re almost there, I promise

Final step is to go back to the “Routes” tab, click on the route which we’ve created earlier and click on Edit button.

In the rule configuration we will enter the hostname which we want to match upon and select our “Service target” from the dropdown.

We can also do advanced request matching based on path, header, method, regex or query string but for now we will use path prefix.

Ok, we have configured our Cloud Load balancer; final step to get our application running is to create a CNAME record for my subdomain “clbtest.slepcevic.net” and point it to “MDEE053110.mesh.akadns.net” (visible in the Summary page of the load balancer).

Let’s go ahead and visit our website. We see that our DeepSparse API server is happily responding and ready to receive requests! Woohoo!

Yes, it’s that easy. In less than 10 minutes we have deployed a globally distributed application on Akamai Connected Cloud. Once IaC support for Cloud Load Balancer is rolled out, we can bring this time down to 5 minutes without any problems.

Ok, 25 regions is cool, but that isn’t truly global is it?

Yes, you’re right; with 25 regions we have covered the large majority of the global (internet) population. Can we do better? For sure! Welcome Gecko!

Gecko?

Akamai’s new initiative, code-named Gecko, is set to revolutionize cloud computing by integrating cloud capabilities directly into Akamai’s extensive edge network. This move aligns perfectly with Akamai’s strategy to provide high-performance, low-latency, and globally scalable solutions. By embedding compute capabilities at or VERY near the edge, Gecko aims to deliver workloads closer to users, devices, and data sources than ever before.

What Does Gecko Mean for Our Deployment in the future?

Gecko’s will enable us to deploy our sentiment analysis application in hundreds of new locations worldwide, including traditionally hard-to-reach areas. This means extremely reduced latency, improved performance, and enhanced availability for users across the world.

The Benefits of Deploying on Gecko

1. Ultra-Low Latency: By running our workloads even closer to end-users, we can drastically reduce the time it takes to process and respond to requests.
2. Global Reach: With Gecko, we can deploy in cities and regions where traditional cloud providers struggle to reach, ensuring a truly global presence.
3. Scalability and Flexibility: With Akamai’s large compute footprint, we can scale out our application to tens of thousands of nodes across hundreds of locations.
4. Consistent Experience: Let’s be real, if you’re running a global application, you’re most probably dealing with multiple providers; with Gecko we can consolidate all of your workloads and location coverage with a single provider. Just the operational benefits of that should be enough to “tickle” your brain into considering it for your application.

Want to try it yourself?

Have fun Go ahead and clone the repo from https://github.com/slepix/neuralmagic-globalLinode and sign up to receive beta access to the new Cloud Load balancer on https://www.linode.com/green-light/ .

NeuralMagic running across all Linode (including Gecko) regions in the next post? Perhaps

Cheers! Alex

The post Deploying a GLOBAL sentiment Analysis service using DeepSparse and Akamai Connected Cloud first appeared on Architect the cloud.

DeepSparse is a sparsity-aware inference runtime that delivers GPU-class performance on commodity CPUs, purely in software, anywhere.

GPUs Are Not Optimal – Machine learning inference has evolved over the years led by GPU advancements. GPUs are fast and powerful, but they can be expensive, have shorter life spans, and require a lot of electricity and cooling.

Other major problems with GPU’s, especially if you’re thinking in the context of Edge computing, is that they can’t be packed as densely and are power ineffective compared to CPU’s; not to mention availability these days.

Since Akamai recently partnered up with Neural Magic, I’ve decided to write a quick tutorial on how to easily get started with running a simple DeepSparse sentiment analysis workload.

In case you want more about Akamai and Neural Magic’s partnership, make sure to watch this excellent video from TFiR. It will also give you a great summary of Akamai’s Project Gecko.

What is Sentiment analysis?

Sentiment analysis (also known as opinion mining or emotion AI) is the use of natural language processing, text analysis, computational linguistics, and biometrics to systematically identify, extract, quantify, and study affective states and subjective information. Sentiment analysis is widely applied to voice of the customer materials such as reviews and survey responses, online and social media, and healthcare materials for applications that range from marketing to customer service to clinical medicine. 

Why is DeepSparse cool? Because I’m doing analysis of 40 thousands movie reviews in 20 minutes using only TWO DUAL CORE Linode VM’s. Mind officially blown.

Let’s do some math here; rounding it up to 120 thousand processed reviews an hour, with 2 instances and a load balancer, we can process over 86 million requests a month which will cost you a staggering 82$ .

If you’re doing that on other cloud providers, you’re paying a five digit monthly bill for that pleasure.

Want to try it yourself? It’s easy!

If you want to try it out on Linode, follow instructions below.

If you want to check out Neural Magic DeepSparse repo, head out here.

Step 1. Clone the Repository.

Open your terminal or command prompt and run the following command:

git clone https://github.com/slepix/neuralmagic-linode

This code will deploy 2 x Dedicated 4 GB virtual machines and a Nodebalancer. It will also install Neural Magic’s DeepSparse runtime as a Linux service and install & configure Nginx to proxy requests to DeepSparse server listening on 127.0.0.1:5543.

WARNING: THIS IS NOT PRODUCTION GRADE SERVER CONFIGURATION!

It’s just a POC! Secure your servers and consult Neural Magic documentation if you want to go to production.

Step 2. – Terraform init

Navigate to the repo using the following command:

cd neuralmagic-linode

If you haven’t already installed Terraform on your machine, you can download it from the official Terraform website and follow the installation instructions for your operating system.

Step 3.

Initialize Terraform by running:

terraform init

Step 4. – Configure your Linode token

Open variables.tf file and paste in your Linode token. If you don’t know how to create a Linode PAT, check this article here. It should look similar like the picture. You can also adjust the region while you’re here

Token in the picture is not valid. It's just an example. 

Step 5 – Run Terraform apply

After configuring your variables, you can apply the Terraform configuration by running:

terraform apply

Terraform will show you a plan of the changes it intends to make.

Review the plan carefully, and if everything looks good, type “yes" and press Enter to apply the changes. Give it 5-6 minutes to finish everything and by visiting your Nodebalancer IP, you should be presented with a landing page for DeepSparse server API.

Step 6.

After the installation is done, it’s finally time to send some data to our API and see how it performs.

We can do that by using curl or invoke-webrequest if you’re on Windows and using Powershell.

CURL:

sentence="Neural Magic & Akamai are cool!"
nodebalancer="172.233.34.110" #PUT YOUR NODEBALANCER IP HERE
curl -X POST http://$nodebalancer/v2/models/sentiment_analysis/infer -H "Content-Type: application/json" -d "{\"sequences\": \"$sentence\"}"

PowerShell:

$sentence = "Neural Magic & Akamai are cool!"
$nodebalancer = "172.233.34.110"

$path = "v2/models/sentiment_analysis/infer"
$api = "http://$nodebalancer/$path"
$body = @{
   sequences = $sentence
} | ConvertTo-Json

(Invoke-WebRequest -Uri $api -Method Post -ContentType "application/json" -Body $body -ErrorAction Stop).content

In both cases make sure to paste in the IP address of the Nodebalancer you deployed and modify the sentence as you wish.

Benchmark time!

In the repository, I’ve included a file called movies.csv and three files; two PowerShell and one Python file.

movies.zip – unzip this one in the same folder where your benchmark scripts are.

analyze.ps1 – PowerShell based benchmark, sends requests in serial – not performant.

panalyze.ps1 – PowerShell based benchmark, sends requests in parallel – better performant

pypanalyze.py – Python based benchmark, sends requests in parallel – best performer (doh!) <-use this

All you need to do to in order to kick off a benchmark is to update the the URL variable with your Nodebalancer IP and you’re off to the races.

Does it scale?

Yes! For kicks I’ve added a third node and the same job finished in 825 seconds. Feel free to add as many nodes as you like and see what numbers you can get. Additionally, you can play with the number of workers in the Python file.

Note 1: python script has been written with the help of ChatGPT :) Results matched with my PowerShell version against verified smaller sample size(check note 2), so I'm gonna call it good :)
	 
Note 2: PowerShell versions don't handle some comments as they should and end up sending garbage to the API. Happens in 3% of the cases. Most probably some encoding/character issue which I couldn't be bothered to fix :)

Note3: Movies.csv file has been generated by using data from https://kaggle.com/

Cheers,

Alex.

The post Sentiment analysis of 40 thousand movie reviews in 20 minutes using Neural Magic’s DeepSparse inference runtime and Linode virtual machines. first appeared on Architect the cloud.