Amazon EC2 F1 Tutorial: Step-by-step guide on running two examples on the Amazon FPGA Cloud

By Jongsok Choi,

  Filed under: Tutorials
  Comments: None

Introduction

Amazon has recently announced the availability of their FPGA cloud, Amazon EC2 F1. We think that this is very exciting news, as it is the first time that FPGAs in the cloud are being available to the general public on a massive scale. This is the first step towards making FPGAs easier to use, as with the EC2 F1, a user no longer has to buy an FPGA and install it on site. FPGAs are typically much more expensive and cumbersome to buy than CPUs or GPUs, hence making them available in the cloud so that one can use them from anywhere around the world makes FPGAs much more accessible.

For more general information about their FPGA instances, visit their website at https://aws.amazon.com/ec2/instance-types/f1/

When the F1 instances first became available, we were excited to try them out, but we found that they were pretty difficult to use at first. Documentation was lacking (and incorrect in some case), and although Amazon provides a few examples with their AWS EC2 FPGA Hardware and Software Development Kit, the instructions to run the examples are scattered in different places and missing some steps. Understandably, they only released this service publicly in April 2017 and documentation may not have been their highest priority. To this end, we decided to write a unified guide which provides step-by-step instructions on how to run the two main examples provided by Amazon, cl_hello_world and cl_dram_dma, on the Amazon EC2 F1. This guide includes the instructions included in their AWS EC2 FPGA Hardware and Software Development Kit as well as information that we have written by trying out the examples ourselves. At the time of writing, we could not find such a step-by-step guide and we ran into issues here and there so we think that this guide will allow one to easily try out the F1 instances without getting stuck in some setup issue. So let’s dive right into it!

In this guide, we will go over two examples:

  1. cl_hello_world: A simple example which writes to some FPGA registers from CPU, then reads them back.
  2. cl_dram_dma: DMA data to FPGA DDR4 memories then DMA them back to verify correctness.

The same steps for running the cl_hello_world can be applied to the cl_dram_dma example, except that for the cl_dram_dma example, you will need to install DMA drivers at the end before executing the software to communicate with the FPGA. Hence we will describe all the steps for the cl_hello_world first then have additional instructions at the end for installing the DMA drivers and running the cl_dram_dma example.

This guide is divided into two parts:

  1. Setting up and synthesizing the example in HDL with Xilinx Vivado
  2. Running the example on an Amazon EC2 F1 instance

Amazon AWS Setup

NOTE: Before starting this tutorial, make sure you can access an F1 instance. If this is the first time you’re trying an FPGA instance with your AWS account, you may need to request an instance limit increase for F1, as the default limit was 0 for our account. The instructions on how to request a limit increase are here. We found this out after spending hours on setting up and synthesizing a design. It took a few business days for the limit increase to be approved.

  • Sign into your AWS console
  • First, you will need to create and download your private key (.pem file). You will need this file to SSH into an AWS instance. The instructions on creating your key can be found here. Make you save this file in a safe place.
  • You will also need to create an access key. This is needed once you’ve SSHed into an instance. To create an access key, go to AWS console, click on the top right corner where it shows your username -> My Security Credentials -> Users -> Click on your user name -> Click on security credentials tab -> Create access key. Once your key has been created, download the access key file, as this is the only time where you can do so. Store it somewhere you can easily find it. If you lose this file, you will have to create a new access key.
  • Launch an FPGA Developer AMI instance from the AWS console. The FPGA Developer AMI comes with all the tools that are needed to use the EC2 F1.   

          -> AWS ‘Services’ -> EC2 -> Launch Instance -> Select ‘AWS Marketplace’ and search for FPGA Developer AMI

  • Select an instance type. For just playing around, use the t2.micro, which is the cheapest instance ($0.012/hour). For actually synthesizing an FPGA project using Vivado, Amazon recommends the more powerful c4.4xlarge instance ($0.796/hour) or the c4.8xlarge instance ($1.591/hour), as even the simple examples can take hours to synthesize. With the c4.4xlarge instance, it took 1 hour to synthesize the cl_hello_world example, 2 hours 40mins for the cl_dram_dma example. 

          -> Click Review and Launch -> Launch

  • Once the instance has been started, SSH into the instance. Click on the “Connect” button to see how to SSH into the instance. Note that the username that shows there is wrong. It should be centos, not root.  (e.g., ssh -i “xxx.pem” centos@ec2-xx-xxx-xxx-xxx.compute-1.amazonaws.com).
  • Once you’ve SSHed in, run “aws configure”. You will need to enter your Access Key ID and Secret Access Key ID, which are in the access key file you downloaded earlier.
  • Clone the aws fpga git repo (git clone https://github.com/aws/aws-fpga.git $AWS_FPGA_REPO_DIR), which contains all SDK, HDK, and all the examples. 
  • CD into the cloned git repo, run “source hdk_setup.sh”. Note that this can take some time (5~10 minutes)

Synthesizing the Example with Xilinx Vivado

  • Change into an example directory and set the CL_DIR environment variable to the path of the example. You will need to set this again if you change examples.
cd $HDK_DIR/cl/examples/cl_hello_world    # you can change cl_hello_world to any other example
export CL_DIR=$(pwd)
  • Verify Vivado is installed.
vivado -mode batch
  • Run Vivado synthesis
cd $CL_DIR/build/scripts
./aws_build_dcp_from_cl.sh

Note that this can take a long time.  If you want to be notified by email when synthesis is done, do this before running the synthesis.

export EMAIL=your.email@example.com
$HDK_COMMON_DIR/scripts/notify_via_sns.py

         For the email to work, you need to set your region name properly during “aws configure”. We set our region name as “us-east-1”.

        Now you can run synthesis with:

./aws_build_dcp_from_cl.sh -notify

        By default, the build runs in the background, but it can be nice to be able to see the synthesis messages to see what’s going on. If you want to run it in foreground:

./aws_build_dcp_from_cl.sh -notify -foreground

Creating an Amazon FPGA Image (AFI)

Now that synthesis is done, we need to create an Amazon FPGA Image (AFI) from the specified design checkpoint (DCP). The AFI contains the FPGA bitstream that will be programmed on the FPGA F1 instance.

  • To create an AFI, the DCP must be stored on S3. So we first need to create an s3 bucket. Make sure your credentials are set up correctly for this (aws configure).
aws s3 mb s3://<bucket-name> --region <region-name> # Create an S3 bucket. Choose a unique bucket name (e.g., aws s3 mb s3://your_awsfpga --region us-east-1
aws s3 mb s3://<bucket-name>/<dcp-folder-name> # Create a folder for your tarball files (e.g.,aws s3 mb s3://your_awsfpga/dcp)
  • Now copy the output files from synthesis to the new s3 bucket.
aws s3 cp $CL_DIR/build/checkpoints/to_aws/*.Developer_CL.tar s3://<bucket-name>/<dcp-folder-name>/
  • Create a folder for your log files          
aws s3 mb s3://<bucket-name>/<logs-folder-name>  # Create a folder to keep your logs
touch LOGS_FILES_GO_HERE.txt                     # Create a temp file
aws s3 cp LOGS_FILES_GO_HERE.txt s3://<bucket-name>/<logs-folder-name>/
  • Copying to s3 bucket may not work if your s3 bucket policy is not set up properly. To set the bucket polity, go to https://s3.console.aws.amazon.com/ -> Click on your bucket -> Click on Permissions tab -> Click on Bucket Policy.
  • Set the policy as listed below, and try copying the files again.
{
    "Version": "2012-10-17",
    "Statement": [
    {
        "Sid": "Bucket level permissions",
        "Effect": "Allow",
        "Principal": {
            "AWS": "arn:aws:iam::365015490807:root"
        },
       "Action": [
           "s3:ListBucket"
        ],
       "Resource": "arn:aws:s3:::<bucket-name>"
    },
    {
        "Sid": "Object read permissions",
        "Effect": "Allow",
        "Principal": {
            "AWS": "arn:aws:iam::365015490807:root"
        },
        "Action": [
            "s3:GetObject"
        ],
        "Resource": "arn:aws:s3:::<bucket-name>/<dcp-folder-name>/*.tar"
    },
    {
        "Sid": "Folder write permissions",
        "Effect": "Allow",
        "Principal": {
            "AWS": "arn:aws:iam::365015490807:root"
        },
        "Action": [
            "s3:PutObject"
        ],
        "Resource": "arn:aws:s3:::<bucket-name>/<logs-folder-name>/*"
    }
    ]
}
  • Verify that the bucket policy grants the required permissions by running the following script:
check_s3_bucket_policy.py --dcp-bucket <bucket-name> --dcp-key <dcp-folder-name>/<tar-file-name> --logs-bucket <bucket-name> --logs-key <logs-folder-name>
  • Once your policy passes the checks, create the Amazon FPGA image (AFI).
aws ec2 create-fpga-image --name <afi-name> --description <afi-description> --input-storage-location Bucket=<dcp-bucket-name>,Key=<path-to-tarball> --logs-storage-location Bucket=<logs-bucket-name>,Key=<path-to-logs>      

       <path-to-tarball> is <dcp-folder-name>/<tar-file-name>

       <path-to-logs> is <logs-folder-name>

       The output of this command includes two identifiers that refer to your AFI: Write these down, as you will need them later.

       – **FPGA Image Identifier** or **AFI ID**: this is the main ID used to manage your AFI through the AWS EC2 CLI commands and AWS SDK APIs.

         This ID is regional, i.e., if an AFI is copied across multiple regions, it will have a different unique AFI ID in each region.  An example AFI ID is **`afi-06d0ffc989feeea2a`**.

      – **Glogal FPGA Image Identifier** or **AGFI ID**: this is a global ID that is used to refer to an AFI from within an F1 instance. For example, to load or clear an AFI from an FPGA slot, you use the AGFI ID.

         Since the AGFI IDs is global (by design), it allows you to copy a combination of AFI/AMI to multiple regions, and they will work without requiring any extra setup. An example AGFI ID is **`agfi-0f0e045f919413242`**.

  • Check if the AFI generation is done. You must provide the **FPGA Image Identifier** returned by `create-fpga-image`:
aws ec2 describe-fpga-images --fpga-image-ids <AFI ID> 

         The AFI can only be loaded to an instance once the AFI generation completes and the AFI state is set to `available`. This can also take some time (Took ~30 minutes for the cl_dram_dma example).

{
    "FpgaImages": [
    {
        ...
        "State": {
            "Code": "available"
        },<
        ...
        "FpgaImageId": "afi-06d0ffc989feeea2a",
        ...
    }
    ]
}
  • Once you have gotten to this point, you have successfully synthesized an HDL design for the EC2 F1. Now you’re ready to program the FPGA and run the example.
  • Go to the EC2 Management Console from AWS console and stop your EC2 instance.

Running the Example on an Amazon EC2 F1 Instance

  • Change your Instance Type to f1.2xlarge (this is the one with an FPGA) and start the instance. To change the instance type:  Right click on your instance shown in the  EC2 Management Console -> Click “Instance Settings” -> Change Instance Type -> Choose “f1.2xlarge”. To start the instance again, don’t click “Launch Instance” as this will create a new instance, but right-click on your instance, “Instance State”, then “Start”.

As mentioned above, if this is the first time you’re trying an F1 instance with your AWS account, you may need to request an instance limit increase.

  • Once the F1 instance is running, SSH into the instance
  • CD into the cloned aws fpga git repo and run “source sdk_setup.sh”
  • Run “aws configure” and input your credentials. If you’ve done this before, and your credentials haven’t changed, you don’t need to do it again.
  • Make sure you clear any AFI you have previously loaded in your slot:
sudo fpga-clear-local-image -S 0
  •  invoke the fpga-describe-local-image command to learn about which AFI, if any, is loaded onto a particular slot. For example, if the slot is cleared (slot 0 in this example), you should get an output similar to the following:
sudo fpga-describe-local-image -S 0 -H

Type  FpgaImageSlot  FpgaImageId     StatusName    StatusCode   ErrorName    ErrorCode     ShVersion
AFI        0            none          cleared          1           ok            0      <shell_version>
Type        FpgaImageSlot  VendorId   DeviceId        DBDF
AFIDEVICE        0          0x1d0f    0x1042      0000:00:0f.0

If the describe returns a status ‘Busy’, the FPGA is still performing the previous operation in the background. Please wait until the status is ‘Cleared’ as above.

  • Now, try loading your AFI to FPGA slot 0:
sudo fpga-load-local-image -S 0 -I <FpgaImageGlobalId>

         <FpgaImageGlobalId> is the ID that you got before when running “aws ec2 create-fpga-image ..” and starts with agfi-….

  • Verify that the AFI was loaded properly. The output shows the FPGA in the “loaded” state after the FPGA image “load” operation. The “-R” option performs a PCI device remove and rescan in order to expose the unique AFI Vendor and Device Id.
sudo fpga-describe-local-image -S 0 -R -H

Type  FpgaImageSlot        FpgaImageId          StatusName    StatusCode   ErrorName    ErrorCode     ShVersion
AFI        0          agfi-0f0e045f919413242     loaded           0           ok            0      <shell version>
Type         FpgaImageSlot  VendorId    DeviceId       DBDF
AFIDEVICE        0           0x6789      0x1d50     0000:00:0f.0
  • Now validate the example. Each CL Example comes with a runtime software under $CL_DIR/software/runtime/ subdirectory. You will need to build the runtime application that matches your loaded AFI.
cd $CL_DIR/software/runtime/ #CL_DIR is hdk/cl/examples/cl_hello_world
make all
sudo ./test_hello_world
  • The cl_hello_world example should show the following output:
AFI PCI  Vendor ID: 0x1d0f, Device ID 0xf000
===== Starting with peek_poke_example =====
register: 0xdeadbeef
Resulting value matched expected value 0xdeadbeef. It worked!
Developers are encouraged to modify the Virtual DIP Switch by calling the linux shell command to demonstrate how AWS FPGA Virtual DIP switches can be used to change a CustomLogic functionality:
$ fpga-set-virtual-dip-switch -S (slot-id) -D (16 digit setting)
In this example, setting a virtual DIP switch to zero clears the corresponding LED, even if the peek-poke example would set it to 1.
For instance:
# fpga-set-virtual-dip-switch -S 0 -D 1111111111111111
# fpga-get-virtual-led  -S 0
FPGA slot id 0 have the following Virtual LED:
1010-1101-1101-1110
# fpga-set-virtual-dip-switch -S 0 -D 0000000000000000
# fpga-get-virtual-led  -S 0
FPGA slot id 0 have the following Virtual LED:
0000-0000-0000-0000
  • As suggested in the output, try changing the Virtual DIP switches:
sudo fpga-set-virtual-dip-switch -S 0 -D 1111111111111111
sudo fpga-get-virtual-led  -S 0

FPGA slot id 0 have the following Virtual LED:
1010-1101-1101-1110

sudo fpga-set-virtual-dip-switch -S 0 -D 0000000000000000
sudo fpga-get-virtual-led  -S 0

FPGA slot id 0 have the following Virtual LED:
0000-0000-0000-0000

sudo fpga-set-virtual-dip-switch -S 0 -D 0000000011111111
sudo fpga-get-virtual-led  -S 0

FPGA slot id 0 have the following Virtual LED:
0000-0000-1101-1110

sudo fpga-set-virtual-dip-switch -S 0 -D 1111111100000000
sudo fpga-get-virtual-led  -S 0

FPGA slot id 0 have the following Virtual LED:
1010-1101-0000-0000
  • Congratulations! You have successfully run your first examples on the EC2 F1!

Running the cl_dram_dma example

  • To run the cl_dram_dma example, follow the same steps describe above to synthesize the HDL, upload the tarball to s3, switch to F1 instance, and program the FPGA.
  • CD into the cl_dram_dma example directory and try running it. $ cd $CL_DIR/software/runtime/(CL_DIR is hdk/cl/examples/cl_dram_dma) $ make all $ sudo ./test_dram_dma
  • If you are running the dma example for the first time, it may not work as you may not have the emda drivers installed.
sudo ./test_dram_dma

Cannot open device file /dev/edma0_queue_0.
Maybe the EDMA driver isn't installed, isn't modified to attach to the PCI ID of your CL, or you're using a device file that doesn't exist?
See the edma_install manual at <aws-fpga>/sdk/linux_kernel_drivers/edma/edma_install.md
Remember that rescanning your FPGA can change the device file.
To remove and re-add your edma driver and reset the device file mappings, run
`sudo rmmod edma-drv && sudo insmod <aws-fpga>/sdk/linux_kernel_drivers/edma/edma-drv.ko`
2017-08-09T17:37:59.877153Z, test_dram_dma, ERROR, test_dram_dma.c +161: dma_example(): unable to open DMA queue. : error_number=2
2017-08-09T17:37:59.877192Z, test_dram_dma, ERROR, test_dram_dma.c +58: main(): DMA example failed: error_number=2
  • To see if the emda is installed, try running:
lsmod | grep edma

          If nothing shows up, that means you need to install the edma drivers           

  • Quick guide on installing the emda drivers:

          Download packages

sudo yum groupinstall "Development tools"
sudo yum install kernel-devel

          Compile the edma driver

cd aws-fpga/sdk/linux_kernel_drivers/edma
make

          Install the driver so it get called everytime the machine boots/reboots

echo 'edma' | sudo tee --append /etc/modules-load.d/edma.conf
sudo cp edma-drv.ko /lib/modules/`uname -r`/
sudo depmod

          To install the driver without rebooting, run

sudo modprobe edma-drv

If you get an error, run dmesg and if you see “edma_drv: exports duplicate symbol xdev_find_by_pdev (owned by xdma)” then disable the xdma driver before enabling edma:

sudo rmmod xdma

Verify that it’s been installed:

lsmod | grep edma 
edma_drv 38830  0

Remove and re-add your edma driver and reset the device file mappings:

sudo rmmod edma-drv && sudo insmod <aws-fpga>/sdk/linux_kernel_drivers/edma/edma-drv.ko

You shouldn’t see any errors.

  • Finally, run the cl_dram_ema example again:

          CD into hdk/cl/examples/cl_dram_dma/software/runtime

          Then run:

sudo ./test_dram_dma
  • You should see the following output.
DRAM DMA read the same string as it wrote on channel 0 (it worked correctly!)
DRAM DMA read the same string as it wrote on channel 1 (it worked correctly!)
DRAM DMA read the same string as it wrote on channel 2 (it worked correctly!)
DRAM DMA read the same string as it wrote on channel 3 (it worked correctly!)
Starting MSI-X Interrupt test
Polling device file: /dev/fpga0_event0 for interrupt events
Triggering MSI-X Interrupt 0
Interrupt present for Interrupt 0. It worked!
  • Congratulations! You’ve now successfully run two examples on the Amazon EC2 F1!

As you probably experienced, it takes a considerable amount of time and effort to use the F1 instances at the moment. We hope that this guide helped in reducing the pain, and we are currently looking to release solutions that will make the F1 instances much easier to use. So stay tuned!

Be the first to write a comment.

Your feedback