Contributed by Sean Greathouse
This project implements a sample api server that satisfies the requirements of the
Test your REST Endpoint URL step in aws-auto-scaling-custom-resource/README.md
It is intended as an example, but can be modified for production use.
- Dockerized API server using Apache and Python
- Scripted setup
- Automated provisioning of SSL certificates from Let's Encrypt
- Demo mode to auto-rotate through application scaling states
- Production mode to enable integration with your custom scaling system
- Docker (tested on Docker CE 18)
- Inbound port 80 & 443 open to the internet
- A valid dns A or CNAME record pointing to your host system
- The host may not run any other process listening on ports 80 or 443
From your host system:
Set the $API_HOME environment variable to the default or your preferred directory.
This variable is required in subsequent commands. Consider setting it as a permanent environment variable.
API_HOME=~/api_home
Clone the repository and build your api_home working directory
git clone git@github.com:aws/aws-auto-scaling-custom-resource.git
cd aws-auto-scaling-custom-resource/sample-api-server
mkdir $API_HOME
cp -Rp api_home/* $API_HOME
cd $API_HOME Pull the Docker base image
docker pull ubuntu:xenialSet environment variables for SSL cert creation
CERTBOT_HOSTNAME must match the dns entry for your host
CERTBOT_EMAIL will be used by Let's Encrypt to notify you of certificate expiration
CERTBOT_EMAIL="foo@example.com"
CERTBOT_HOSTNAME="api.example.com" Docker run to populate the directories in api_home with apache config and ssl certificates.
All system config, state, and logs are stored in these directories on your local host.
docker run --rm -p 80:80 -it --name api_build_static \
-v "$API_HOME"/var/www:/var/www -v "$API_HOME"/var/log/apache2:/var/log/apache2 \
-v "$API_HOME"/etc/apache2:/etc/apache2 -v "$API_HOME"/usr/lib/cgi-bin:/usr/lib/cgi-bin \
-v "$API_HOME"/etc/letsencrypt:/etc/letsencrypt -v "$API_HOME"/scripts:/root/scripts \
--env CERTBOT_EMAIL="$CERTBOT_EMAIL" --env CERTBOT_HOSTNAME="$CERTBOT_HOSTNAME" \
ubuntu:xenial /root/scripts/setup.shBuild the Docker image for the api server
docker build $API_HOME/ -t api_image
Run the Docker image as a daemonized container
- Listening on ports 80 & 443
- Mounting config, docroot, log and cgi directories from localhost over the same directories in the Docker container.
Note that the container will not run automatically at system startup.
docker run --rm -p 80:80 -p 443:443 -d --name api \
-v "$API_HOME"/var/www:/var/www -v "$API_HOME"/var/log/apache2:/var/log/apache2 \
-v "$API_HOME"/etc/apache2:/etc/apache2 -v "$API_HOME"/usr/lib/cgi-bin:/usr/lib/cgi-bin \
-v "$API_HOME"/etc/letsencrypt:/etc/letsencrypt \
api_imageTo test your installation call the following url from a browser (replacing the hostname with your own)
https://api.example.com/v1/scalableTargetDimensions/1-23456789
you should see a json response:
{"scalableTargetDimensionId": "10", "scalingStatus": "Successful", "resourceName": "MyService", "desiredCapacity": 1.0, "actualCapacity": 1.0, "dimensionName": "MyDimension", "version": "MyVersion"}
If the above fails, test without https
http://api.example.com/v1/scalableTargetDimensions/1-23456789
You can also test from the command line of your host
curl http://localhost/v1/scalableTargetDimensions/1-23456789
If that fails, you can test to see if the Docker container is running
docker ps
And if Apache is working on port 80
curl localhost
Once you require host certificates for your api backend, you will no longer be able to test your api over https.
If your outbound network allows unencrypted PATCH commands you can test over http.
However, the following setup is not required for the integration to work so you can leave this step until after testing the full system.
Get the certificate file you pulled pulled from API Gateway in the
Configure SSL/HTTPS section of aws-auto-scaling-custom-resource
The certificate should look like the following (with more text between the BEGIN and END lines):
-----BEGIN CERTIFICATE-----
MIIC6TCCAdGgAwIBAgIJAKumXi6NTmSBMA0GCSqGSIb3DQEBCwUAMDQxCzAJBgNV
BAYTAlVTMRAwDgYDVQQHEwdTZWF0dGxlMRMwEQYDVQQDEwpBcGlHYXRld2F5MB4X
EcQhaxYT710cDFtf9kkbzTQMt0os4mKuItILvKQ.......
-----END CERTIFICATE-----
Edit the client certificate public key file and paste in the certificate
vim $API_HOME/etc/apache2/public-keys/ProdClientCertificate.pem
To require host certificate verification edit the Apache config file for the HTTPS Virtual Host
vim $API_HOME/etc/apache2/sites-available/0-api.conf
un-comment the following lines and save the file
#SSLVerifyClient none
#SSLCACertificateFile "public-keys/ProdClientCertificate.pem"
#<Location "/v1/scalableTargetDimensions/">
#SSLVerifyClient require
#SSLVerifyDepth 1
#</Location>
then restart Apache in the container
docker exec api /usr/sbin/apachectl restart
When AWS Auto Scaling calls the API to initiate a scaling event it issues an https PATCH call with the new desired capacity.
To test directly in Postman, send a PATCH request to:
http://api.example.com/v1/scalableTargetDimensions/1-23456789
with a body payload of:
{"desiredCapacity":2.0}
The 1-23456789 in the URL above can be replaced by any ID. The system can manage separate state for an arbitrary number of IDs.
AWS Auto Scaling will use the ID you set in the Register a Scalable Target section of aws-auto-scaling-custom-resource
The system will respond by setting the desiredCapacity to 2.0 and scalingStatus to Pending
{"scalingStatus": "Pending", "scalableTargetDimensionId": "1-23456789", "version": "MyVersion", "resourceName": "MyService", "actualCapacity": 1.0, "desiredCapacity": 2.0, "dimensionName": "MyDimension"}
By default the system will increment the state for the scalableTargetDimensionId with each subsequent GET.
- "scalingStatus": "InProgress"
- "scalingStatus": "Successful" - "actualCapacity": 2.0
You can see the event progression in the api log (replace the 20180822 with the current host date):
tail -f $API_HOME/var/log/apache2/api-20180822
20180822-234725 Request: PATCH dimensionId: 1-23456789 {"desiredCapacity": 2.0}
20180822-234725 Response: dimensionId: 1-23456789 desiredCapacity: 1.0 -> 2.0 scalingStatus: Successful -> Pending
20180822-235128 Request: GET dimensionId: 1-23456789
20180822-235128 Response: dimensionId: 1-23456789 scalingStatus: Pending -> InProgress
20180822-235303 Request: GET dimensionId: 1-23456789
20180822-235303 Response: dimensionId: 1-23456789 actualCapacity: 1.0 -> 2.0 scalingStatus: InProgress -> Successful
Now that you have verified that the backend API works, you can proceed with the Test the Scaling Policy section in aws-auto-scaling-custom-resource and watch the AWS Auto Scaling service issue scaling commands in the API log.
If you want to simulate a scaling failure, change the cgi scripts testFailure mode.
testFailure works in demoMode by setting scalingStatus to Failed instead of Successful in the final scaling stage.
vim $API_HOME/usr/lib/cgi-bin/api.py
testFailure = True
In a production scaling scenario, the system that does the actual application scaling will need to update the scaling state in the API. In production mode the cgi script will still set scalingStatus from Successful to Pending as a response to the initial desiredCapacity change issued by AWS Auto Scaling.
To set production mode:
vim $API_HOME/usr/lib/cgi-bin/api.py
demoMode = False
The state of each scalableTargetDimensionId can be set by issuing a PATCH to the API with the appropriate json body payload.
{"scalingStatus": "Successful", "actualCapacity": 2.0}
The current implementation saves state into files in $API_HOME/var/www/state/ on the local host.
For an actual production system, state should be managed in a highly available data store accessible from multiple hosts.
This change could be as simple as saving state to an NFS system, or by re-writing the write_state and read_state functions in api.py to talk to a NoSql key-value store.
The following steps close security holes that are intentionally open for dev and test. Follow your own security best practices if you deploy this system.
Disable port 80 on one or more layers of the system
- Docker layer
Remove -p 80:80 from the docker run statement - Apache layer
Disable the Apache VirtualHosts that listen on 80
docker exec api /usr/sbin/a2dissite 1-api-80.conf
docker exec api /usr/sbin/a2dissite 3-api-internal-80.conf
docker exec api /usr/sbin/apachectl restart - Block external port 80 traffic in your firewall or AWS VPC Security Group
Disable access to docroot by editing the relevant virtual host conf files
$API_HOME/etc/apache2/sites-available/0-api.conf
$API_HOME/etc/apache2/sites-available/1-api-80.conf
$API_HOME/etc/apache2/sites-available/2-api-internal.conf
$API_HOME/etc/apache2/sites-available/3-api-internal-80.conf
and un-commenting the following text block
#<Directory "/var/www/html/">
# Options FollowSymLinks
# AllowOverride None
# Order Deny,Allow
# Deny from All
#</Directory>
Require a host certificate as documented above
Apache only supports a single host certificate per VirtualHost so enabling host certificates in 0-api.conf will block all other https client acccess.
You can continue to access the api by configuring an alternate dns ServerName in the 2-api-internal.conf or 3-api-internal-80.conf conf files. To use https for internal api access you will need to set up valid SSL certificates.
To auto-renew the Let's Encrypt certificate, run the following as a scheduled command on your host.
docker exec api /usr/bin/certbot renew