We were invited to publish our AWS F1 Memcached acceleration work at the 2018 International Symposium on Highly-Efficient Accelerators and Reconfigurable Technologies (HEART). Our paper describes in detail the Memcached accelerator on AWS F1 as well as many of the interesting things (good and bad) that we learned about the F1 infrastructure along the way.
J. Choi, R. Lian, Z. Li, A. Canis, J. Anderson, “Accelerating Memcached on AWS Cloud FPGAs”, HEART 2018 (PDF)
We are excited to release a live demo of our Memcached server accelerator on AWS F1.
In the demo, we will automatically spin up two AWS EC2 instances for you (for free) so that you can easily try out our FPGA Memcached server accelerator. An F1 instance is programmed as the Memcached server and an M4 CPU instance is used as the client to run the memtier_benchmark. Note that starting the two instances can take a few minutes, so please be patient if the demo page takes some time to load.
As shown below, the demo page shows two terminal windows, one for the client and another for the server. In the client window, you will be able to choose the number of connections and the number of requests to the Memcached server to start the benchmark. While a test is running, the server window shows the packets-per-second (PPS) received and sent by the F1 instance. As described in our previous blog post, 700K is the maximum PPS on F1. When the test finishes, the client windows shows the measured throughput and latency to the Memcached server. For the demo, we use 100-byte data values, 1:1 set/get ratio, and Memcached pipelining of 16.
We attended RedisConf last week to learn more about Redis and meet users who are using Redis/Memcached. Many people were interested in our 11M+ Memcached ops/sec result achieved with a single AWS EC2 F1 instance, but the 1-page summary document was not comprehensive enough to answer all the questions. Here we explain in more detail about the hardware architecture and how we set up the experiment.
We had described the architecture in a previous blog post, but we briefly review it here. As shown in the figure below, the entire TCP/IP (and UDP) network stack, as well as the Memcached server logic, is implemented in FPGA hardware. We deeply pipeline the FPGA hardware (if this phrase isn’t clear, please see “What is FPGA Pipelining?” at the end of this post) to process many network packets and Memcached requests in flight. This cannot be done on a CPU, hence we are able to achieve a big speedup vs. CPU Memcached servers. On the F1, the FPGA is not directly connected to the network, so we use the CPU to transfer incoming network packets directly from the NIC to the FPGA and also outgoing network packets from the FPGA back to the NIC. Note that the Memcached accelerator is a prototype and currently only supports get and set commands.
We are pleased to present the world’s fastest cloud-hosted Memcached on AWS using EC2 F1 (FPGA) instances. With a single F1 instance, LegUp’s Memcached server prototype achieves over 11M ops/sec, a 9X improvement over ElastiCache, at <300 μs latency. It offers 10X better throughput/$ and up to 9X lower latency compared to ElastiCache. Please refer to our 1-page handout for more details.
On our last blog post, we wrote about using LegUp to perform networking processing on AWS cloud FPGAs (F1). In this post, we describe accelerating Memcached on AWS F1.
Memcached is a high-performance distributed in-memory key-value store, widely deployed by companies such as Flickr, Wikipedia and WordPress. Memcached is typically used to speed up dynamic web applications by caching chunks of data (strings, objects) in RAM, which alleviates the load on the back-end database.
The figure below shows a typically deployment where Memcached is used as a caching layer to provide fast accesses to data for front-end web servers. When the data is found on a Memcached server, trips to disk storage are avoided (i.e. disk stores attached to the back-end databases). Memcached is also used by Facebook, Twitter, Reddit, and Youtube.
As network bandwidths continue to increase from 10 Gbps to 25Gbps and beyond, cloud users need to be able to process high-bandwidth network traffic intelligently, to perform real-time computation or to gain insight into network traffic for detecting security or service issues.
We live in an exciting time, with FPGA cloud instances now available from Amazon and Alibaba. Traditionally, FPGAs (field programmable gate arrays) have been used in network switches or for high-frequency financial trading because of their superior ability to perform high-bandwidth network processing. We believe many cloud applications that require high-speed network and data stream processing can achieve 10X better latency and throughput on a cloud FPGA compared to a standard commodity server.
LegUp Cloud FPGA Platform
We have developed a cloud FPGA platform that makes it easier for a software developer trying to program high-speed data processing on a cloud FPGA. Behind the scenes, the LegUp platform hides all the low-level details of getting network traffic to/from the FPGA and handling the network layer. We currently support AWS F1 FPGA instances.
TORONTO, February 22, 2018 — LegUp Computing, Inc. announced today that it closed a seed funding round led by Intel Capital. LegUp offers a cloud platform that enables software developers to program, deploy, scale, and manage FPGA devices for accelerating high performance applications without requiring hardware expertise. The technology enables the next generation of low-latency and high-throughput computing on the vast amount of real-time data processed in the cloud. LegUp Computing, Inc., was spawned from years of research in the Dept. of Electrical and Computer Engineering at the University of Toronto to commercialize the award-winning open-source LegUp high-level synthesis tool.
LegUp Computing team from left to right: Omar Ragheb, Zhi Li, Dr. Andrew Canis, Ruolong Lian, Dr. Jongsok Choi, and University of Toronto Professor Jason Anderson (Photo: Jessica MacInnis)
The quality and price of image sensors has seen a huge improvement over the past decade, we are now seeing increased adoption of cameras in the automotive sector. One new application is a driver facing camera that can monitor the driver for signs of drowsiness. If the driver is about to fall asleep, we can trigger an alarm. Implementing a system like this requires a camera and an embedded processor to analyze the video stream, looking for the driver’s face and performing facial landmark detection to determine the location of their eye lids.
We have recently worked with the company Eyeris who specializes in these facial analytics software algorithms. However, they were having a problem, the software algorithms ran too slowly on an embedded processor. They came to Efinix, a company that specializes in programmable hardware acceleration platforms, who contacted us to help them convert this facial analysis written in software into hardware that can run on an FPGA.
In the video below shows three versions of the facial analytics demo. First, running on the embedded processor (~5 frames per second), then running on an FPGA (~13 FPS), and finally on a smaller video canvas (~15 FPS). You can see that the responsiveness improves tremendously by using the FPGA to accelerate this application. Eyeris showed this demo to some of their customers during CES this year:
In this post we’re going to show you a video filtering demonstration on an Amazon cloud FPGA using LegUp. The specific video filter we will showcase is Canny edge detection as we described in a previous post. This same approach could be used to implement any other image filter that uses convolution (blur, sharpen, emboss, etc.).
We’re from Toronto and fans of the Blue Jays, so we’ll use a slow motion video of Josh Donaldson hitting a home run to demonstrate our filter:
A paper describing the use of LegUp HLS to synthesize hardware cores for floating-point computations from C-language software will appear in the 2018 ACM/IEEE Design Automation and Test in Europe (DATE) conference, to be held at Dresden, Germany, in March 2018. The floating-point cores are fully IEEE 754 compliant, yet through software changes alone, can be tailored to application needs, for example, by reducing precision or eliminating exception checking, saving area and raising performance in non-compliant variants. The IEEE-compliant cores synthesized by LegUp HLS compare favourably to custom RTL cores from FloPoCo and Altera/Intel, despite the LegUp-generated cores being synthesized entirely from software. An advance copy of the paper, jointly authored by the University of Toronto and Altera/Intel, is available: PDF.