James Ridgway - Software Engineer

Build your own custom Short URL generator using AWS

Use terraform to quickly setup your own Short URL generator using a custom domain with AWS API Gateway, CloudFront, Lambda, Route 53 and S3.

I wanted short URLs that used a custom domain and I didn't want to pay a lot for it. If I went the "do it yourself" route I also didn't want anything that would be a maintenance burden. I've found that using AWS API Gateway, CloudFront, Lambda, Route 53 and S3 can be used to do this cheaply. And if I'm being honest I've wanted to experiment with AWS Lambda for a while, and this was a good excuse.

Introducing the AWS Services

AWS provides a lot of cloud services. Some of them have are rather creatively (and uninituitively named). I'll provide a brief introduction to the services I used to build the Short URL generator.

  • API Gateway
    Allows you to host a HTTP API that can proxy requests to other services, such as other HTTP service, AWS Lambda functions, etc. API Gateway also supports authentication and API key management.
  • CloudFront
    Amazon's content delivery network.
  • Lambda
    AWS Lambda lets you run small snippets of code without provisioning a server. Lambda functions can be triggered by a variety of AWS services, and can use the AWS APIs.
  • Route 53
    Amazon's DNS.
  • S3
    Blob (file) storage at it's simplest. Supports additional features such as hosting statis websites.

If you want to understand the naming of more AWS service, check out ExpeditedSSL article on Amazon Web Services in Plain English.

The Approach

The plan is to use CloudFront to cache redirecting web pages at the edge of the CloudFront network that will redirect form the short URL to the full URL.

The redirecting web pages will be served up from S3. With S3 you can create an object with a meta data entry called Website Redirect Location. When an S3 bucket is configured to host a static website objects ( with a Website Redirect Location metadata entry) will be served up over HTTP as a redirecting webpage.

Short URLs - Project Approach

API Gateway and AWS Lambda will be used to create and delete shortlinks via HTTP API calls. The API will be protected with an API key, and will be served up via the same CloudFront distrubtion.

Finally Route 53 will alias the custom domain name to the domain name of the CloudFront distribution.

Terraform

Terraform is an open source tool for configuring and provisioning infrastructure from code. Terraform supports a number of cloud providers including AWS, Google Cloud Platform, Azure, Digital Occean and OpenStack.

I've opted to develop this project using Terraform so that I can easily create the setup in a reproducible way. This will also allow others to download the code and setup their own short URL generator in minutes.

Step by Step

Although you'll be able to read how the solution works from the Terraform code, here is a high level step by step explanation of the approach.

  1. Create an S3 bucket with static website hosting enabled
    This bucket will serve up empty objects where the key is the short URL suffix.
  2. Write the AWS Lambda functions
    There will need to be two lambda functions that can take an API Gateway request, process it and return a response. There will be one function for creating a short URL and another for deleting a short URL
  3. Create the API with AWS API Gateway
    The API will have one endpoint for creating a short URL and another for deleting a short URL. Both endpoints will call out to their respective AWS Lambda functions.
  4. CloudFront Distribution
    Create a single CloudFront distribution with the S3 bucket and the API Gateway configured as origin. Using behaviour rules we will control whether a request is routed to the S3 bucket or the API Gateway.
  5. Update Route 53
    Update Route 53 to alias the custom domain to the CloudFront distribution domain name.

The API Gateway will need the necesary permissions to execute the lambda functions and the lambda functions will need permission to create and delete objects in the S3 bucket.

Deploying With Terraform

You can find the full terraform code on GitHub at github.com/jamesridgway/aws-lambda-short-url.

Hosted Zone Setup

The domain you want to use for short URLs needs to be setup in Route 53 before applying the terraform state.

  1. In Route 53 click Create Hosted Zone
  2. Create the domain as a Public Hosted Zone
  3. Setup the DNS Nameservers for your domain to use Route 53 (nameservers are listed in the NS record).
    Route 53 - DNS

Initialise Terraform

I've setup terraform to use S3 as a backend for tracking the state. By default terraform will track the state of operations on the local file system, however using a remote backend can be useful if:

  • You're working in a team
  • You're working across multiple computers

Start by cloning the repository:

$ git clone https://github.com/jamesridgway/aws-lambda-short-url.git
$ cd aws-lambda-short-url

Initialise the backend to use an S3 bucket to store the state (this only needs to be done once):

$ terraform init -backend-config "bucket=terraform-states.example.com"

Alternatively you can remove terraform.tf which defines the backend store - this will cause terraform to default to local file storage.

Applying the Terraform State

Use apply to apply the state described in the terraform files:

$ terraform apply

You'll be prompted for:

  • the custom domain you want to use
  • your AWS account ID.

Once the setup has completed you'll see an output similar to the following:

Outputs:

Admin API Key = uWyv6B1NPI0vWxVPeQD46ctlmWd6l7x3YLSYCRf0
CloudFront Domain Name = d111111abcdef8.cloudfront.net
Short URL Doamin = example.com

Note: you will need to wait for the CloudFront distribution to be full initialised (the Status will change to Deployed). This can take ~20 minutes.

Using the API

Deploying the infrastructure with terraform will take only a few minutes and once the CloudFront distribution has been fully initialised you'll be ready to start creating URLs.

Creating a Short URL

Creating a URL is done with a POST request to the /admin endpoint. The x-api-key header should be set to the Admin API Key value that was generated in the output of the terraform setup:

curl -X POST \
    -d '{"url": "https://www.james-ridgway.co.uk/blog/build-your-own-custom-short-url-generator-using-aws"}' \
    -H "x-api-key: XXXXX" \
    http://jmsr.io/admin

The response will provide you with the full short URL and token value in JSON output:

{
    "short_url": "https://jmsr.io/cwM1iQ",
    "url": "https://www.james-ridgway.co.uk/blog/build-your-own-custom-short-url-generator-using-aws",
    "token": "cwM1iQ"
}

Visit a Short URL

So here's one of my short URL: https://jmsr.io/cwM1iQ. This link is a short link to this blog post.

CloudFront serves up the empty S3 object as shown below using CURL with the vebose flag. You get a 301 Moved Permanently response with the Location header set to the full URL.

$ curl -v https://jmsr.io/cwM1iQ
*   Trying 54.192.197.16...
* Connected to jmsr.io (54.192.197.16) port 443 (#0)
* found 148 certificates in /etc/ssl/certs/ca-certificates.crt
* found 597 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
*        server certificate verification OK
*        server certificate status verification SKIPPED
*        common name: jmsr.io (matched)
*        server certificate expiration date OK
*        server certificate activation date OK
*        certificate public key: RSA
*        certificate version: #3
*        subject: CN=jmsr.io
*        start date: Mon, 07 May 2018 00:00:00 GMT
*        expire date: Fri, 07 Jun 2019 12:00:00 GMT
*        issuer: C=US,O=Amazon,OU=Server CA 1B,CN=Amazon
*        compression: NULL
* ALPN, server accepted to use http/1.1
> GET /cwM1iQ HTTP/1.1
> Host: jmsr.io
> User-Agent: curl/7.47.0
> Accept: */*
> 
< HTTP/1.1 301 Moved Permanently
< Content-Length: 0
< Connection: keep-alive
< Date: Sat, 12 May 2018 08:36:48 GMT
< Location: https://www.james-ridgway.co.uk/blog/build-your-own-custom-short-url-generator-using-aws
< Server: AmazonS3
< X-Cache: Miss from cloudfront
< Via: 1.1 95a4581bed116b6338fc42595fee6f43.cloudfront.net (CloudFront)
< X-Amz-Cf-Id: s4q4k2DkWwc6jxNvA2XWQYK_wJC51_QDag2CucX-a67aq3si78_9Gw==
< 
* Connection #0 to host jmsr.io left intact

Deleting a Short URL

Deleting an endpoint is also done via a DELETE request to /admin/<token>, for example:

curl -X DELETE -H "x-api-key: XXXXX" http://jmsr.io/admin/cwM1iQ

The End Result

Using AWS has allowed me to produce a short URL service with a custom domain that has little maintenance overhead. The short URLs are served up via CloudFront and S3 which are both cost effecitve services.

This solution also allows me to create and delete short URLs by simple CURL commands.

If you want your own custom short URL service, you can use the terraform code I've shared to get setup in minutes:

  1. Purchase a domain
  2. Create the domain in Route 53 and update nameservers
  3. Checkout aws-lambda-short-url
  4. Initialise and apply the terraform state

Enjoy!

Read More