Continuous Integration and Deployment with Drone, Docker, Django, Gunicorn and Nginx - Part 2
Recently updated on
The Introduction
This is the second part of a multi-part tutorial covering a simple(ish) setup of a continuous integration/deployment pipeline using Drone.io. Since Part 1, I’ve added a GitHub project outlining a simple Django application that you can use as a reference.
In Part 2, we will be adding a publish
step to our application’s drone.yml
in order to push our image to Docker Hub. Having our image in Docker Hub allows us to easily pull our app’s image onto our staging/production server to allow easy automated deployment. After the publish
step we will add a deploy
step to the drone.yml
that will be responsible for SSHing into an EC2 instance (or wherever your app lives) to pull our newly pushed image and update our app.
Let’s get started!
Step 1: Publish our app’s image to Docker Hub after a successful pull request.
If you recall, our drone.yml should successfully be running our application’s test suite on push and pull events and reporting back to GitHub if our test suite failed or succeeded. After a successful build, we would like to push our app’s Docker image to Docker Hub.
Sign up/Log in to Docker Hub
Having an account on Docker Hub is free and it will allow us to easily update our application once it is on our EC2 instance.
After you have set up an account, keep note of your username and password. We will need to inject those as Drone secrets in order for Drone to push our app’s image. Read more about pushing images to Docker Hub.
In your .drone.yml
, add a “publish” section so that it looks like this:
pipeline:
build:
image: python:3.5.2
environment:
- DATABASE_URL=postgres://postgres@localhost
commands:
- sleep 5
- pip3 install -r requirements.txt # make sure gunicorn is installed
- cd projectDir
- python ./manage.py test
- cd ..
when:
branch: [ master, develop ]
event: [push, pull_request ] # trigger step on push and pull events
publish:
image: plugins/docker
username: ${DOCKER_USERNAME} # we will inject your dockerhub username using drone secrets.
password: ${DOCKER_PASSWORD} # we will inject your dockerhub password using drone secrets.
email: octocat@catmail.com
repo: octocat/repoName # refer to dockerhub documentation for repo naming conventions
tag: latest
file: Dockerfile
environment:
- DOCKER_LAUNCH_DEBUG=true #( usefull for debugging but not necessary )
when:
branch: [ master ]
event: [ push ] # step only triggers on push events
services:
database:
image: postgres
environment:
- DATABASE_URL=postgres://postgres@localhost
We are telling Drone to use the plugins/docker
image (read more about this particular plugin or plugins in general). In order for the plugin to login to your Docker Hub account so it can push your image it will need your username and password passed in as Drone secrets.
On your local machine (assuming the drone CLI
is setup correctly in Part 1 ), run:
$ drone secret add --image=plugins/docker octocat/repoName DOCKER_USERNAME yourDockerUsername
$ drone secret add --image=plugins/docker octocat/repoName DOCKER_PASSWORD yourDockerPassword
To make sure your secrets have been set correctly, you can run:
$ drone secret ls octocat/repoName
and you should see something like:
DOCKER_USERNAME
Events: push, tag, deployment
SkipVerify: false
Conceal: false
DOCKER_PASSWORD
Events: push, tag, deployment
SkipVerify: false
Conceal: false
Now that we are using secrets, you must sign your project by running the following command inside your projects folder (so that the Drone CLI will create a .drone.yml.sig
file next to your .drone.yml
file:
$ drone sign octocat/repoName
Now, every time you change your .drone.yml
locally, you will have to run the sign command to generate a new sig
file. If you don’t, then the next Drone build will not have access to your secrets. Drone is pretty good at displaying a warning in the console of the build if this is the case.
Go ahead and open another pull-request. After the tests pass and you’ve merged your new code in, you should see Drone run another build but this time executing the new publish
step we outlined in the drone.yml
. If problems occur you should at least see some helpful output thanks to the DOCKER_LAUNCH_DEBUG=true
line in your .drone.yml
. Drone secrets and plugins can be fickle. I’ve found that even if you do not explicitly declare environment variables in your .drone.yml
all secrets you’ve set for that plugin and repo are passed into the resulting Docker container pertaining to that build step. So if you were to remove the following two lines from your .drone.yml
file:
username: ${DOCKER_USERNAME} # we will inject your dockerhub username using drone secrets.
password: ${DOCKER_PASSWORD} # we will inject your dockerhub password using drone secrets.
Your Docker username
and password
would still be passed into the publish step’s Docker container, resulting in a successful Docker Hub login. This can lead to some odd behavior with certain plugins (this phenomenon might be specifically plugin-dependant so I encourage you to read all plugin documentation).
If the build passes, go ahead and visit your Docker Hub account. You should see a newly pushed image as a result! We’re almost there!
Step 2: Create a service to start your app’s Docker container
If you already have a service to run your Django application inside a Docker container, you can skip this step.
Here you have two choices. Either spin up a new EC2 container and install Docker (which will you cost you a small amount of money with two running free-tier instances) or you can execute the below steps on the same EC2 instance that you put your Drone server/agent on (though I have not tested this and the exact steps might deviate). SSH into the instance you want you app to be hosted on.
Create an Init Script for your Django Application
This example uses systemd, but there are some alternatives such as upstart. Create a service script as follows:
$ vim /etc/systemd/system/projectName.service
[Unit]
Description=yourApp Container
Requires=docker.service
After=docker.service
[Service]
Restart=always
ExecStart=/usr/bin/docker run --name=containerName -p 8000:8000 octocat/repoName bash -c "gunicorn smashDB.wsgi -b 0.0.0.0:8000"
ExecStop=/usr/bin/docker stop -t 2 repoName
ExecStopPost=/usr/bin/docker rm -f repoName
[Install]
WantedBy=default.target
Here we are assuming Gunicorn is installed in your app. The first -p 8000:8000
is telling all traffic on port 8000 that reaches our EC2 container to be forwarded to the created Docker container on port 8000. This allows traffic from 0.0.0.0
to be translated so Django can process them. We’ve set the Restart
flag to always
so that if the container goes down for whatever reason, a new one will take it’s place. The ExecStop
and ExecStopPost
command here just tell systemd to clean up after itself.
To start using the service, reload systemd and start the service:
systemctl daemon-reload
systemctl start projectName.service
If you want the service execute whenever your EC2 instance starts, run:
systemctl enable docker-redis_server.service
Now if you run:
sudo docker ps
you should see a Docker container running your Django app.
Create a deploy.sh
file
Now that our container is running, we can create a sh
file that will be responsible for stopping our service, stopping and removing our app’s Docker container, pulling our app’s latest image from Docker Hub, and finally restarting our service. Create a deploy.sh
file (I created mine in ~/deploy/)
. Your deploy.sh
file should look something like this:
#!/bin/bash
echo "Updating staging Server"
echo "stopping projectName.service"
sudo systemctl stop projectName.service
# remove all outdated images and containers
echo "removing outdated/dangling images and containers"
sudo docker rm $(sudo docker ps -aq)
sudo docker rmi $(sudo docker images --filter dangling=true --quiet)
# pull new image for projectName
echo "pulling new image for myProject"
sudo docker pull octocat/repoName
# restart service which will use the newly pulled image
echo "restarting projectName service"
sudo systemctl start projectName.service
# App is updated!
echo "projectName successfuly updated!"
Make the file executable
sudo chmod +x deploy.sh
and execute manually just to make sure it is working:
sh deploy.sh
You should see the echo statements and if you run:
docker ps
you should see that your app was just newly created
. With your service and deploy.sh
we can now add a publish step to our
drone.yml!
Step 3: Add a deploy
step to your drone.yml
In your drone.yml
, add a ssh-deploy step so that it looks like this:
pipeline:
build:
image: python:3.5.2
environ ment:
- DATABASE_URL=postgres://postgres@localhost
commands:
- sleep 5
- pip3 install -r requirements.txt # make sure gunicorn is installed
- cd projectDir
- python ./manage.py test
- cd ..
when:
branch: [ master, develop ]
event: [push, pull_request ] # trigger step on push and pull events
publish:
image: plugins/docker
username: $DOCKER_USERNAME # we will inject your dockerhub username using drone secrets.
password: $DOCKER_PASSWORD # we will inject your dockerhub password using drone secrets.
email: octocat@catmail.com
repo: octocat/repoName
tag: latest
file: Dockerfile
environment:
- DOCKER_LAUNCH_DEBUG=true #( usefull for debugging but not necessary )
when:
branch: [ master ]
event: [ push ] # step only triggers on push events
ssh-deploy:
image: appleboy/drone-ssh
pull: true # always pull the latest version of the `drone-ssh` plugin
host: ${HOST} # passed in as a drone secret
user: ${USER} # passed in as a drone secret
key: ${SSH_KEY} # passed in as a drone secret
port: 22
pull: true
command_timeout: 180
script:
- cd /home/ubuntu/deploy # or whereever you put your `deploy.sh`
- sh deploy.sh
when:
event: [push, tag, deployment]
services:
database:
image: postgres
environment:
- DATABASE_URL=postgres://postgres@localhost
The $HOST
, $USER
and $SSH_KEY
variables will be passed in as Drone secrets. Read up on the appleboy/drone-ssh
plugin.
Set your secrets:
drone secret add --image=applyboy/drone-ssh octocat/repoName HOST yourEC2InstancePublicDNS
drone secret add --image=applyboy/drone-ssh octocat/repoName USER ubuntu
drone secret add --image=applyboy/drone-ssh octocat/repoName SSH_KEY @/path/to/pem/key.pem
Remember to resign your drone.yml
drone sign ocotcat/repoName
Open up one last pull request. When your build passes merge in your latest code. Upon merge you should see the deploy step trigger (along with all of the output from your deploy.sh
file) after the publish step in the Drone console. Congratulations! We now have a very simple CI/CD pipeline.
In Part 3 I will outline setting up Nginx for your app and add a slight tweak to the deploy.sh
to accomodate the containerized nature of your app with Nginx.