Integrating migrations in a continuous delivery pipeline with CircleCI
To Skip the tutorial and navigate to the code, see the example repo on GitHub.
Introduction
Like handling changes to a database, integrating Contentful into a continuous integration (CI) pipeline enables changes to a content model and migrating data as part of a deployment process. Further, it enables testing content changes and canceling a deploy process thereby preventing interruptions or downtime, in case the tests fail.
Using the Contentful CLI's space migration command, it is possible to create, delete, and edit content models and entries programmatically entirely in JavaScript. By using the Migrations DSL tool over the Contentful web app, updating a content model becomes a repeatable process. Changes made to a content model can also be tracked by integrating it to the version control.
Space environments are entities within a space that allow a developer to create and maintain multiple versions of space-specific data, and make changes to them in isolation. This promotes teams to work on modifying a Contentful space, as any changes made to an environment do not affect production data. Every time you create a new environment, you're creating a copy from the current version of your master space. Contentful migration tooling, when combined with space environments, starts integrating Contentful into the CI Pipeline.
A typical continuous integration pipeline creates a build and runs tests against it, before deploying code. Integrating content migrations into that pipeline requires additional steps.
In the build phase, add a step to programmatically add a new environment, while running any needed migrations as part of deployment on that new environment. During the testing phase, utilize the newly-migrated environment. Assuming everything passes, we'll create a new environment, run those migrations and update our master alias to target this created environment, and then deploy the code into production.
Pre-requisites
- Understanding of using the Contentful CLI to write migration scripts, Space Environments and using Environment Aliases to update a master alias, with one already setup for this project.
- A CircleCI account, but it’s possible to modify the project to work with Jenkins, Travis, Buildbot or any other Continuous Integration software.
- A GitHub to host your code and Heroku for the deployment stage of this project.
Setting up Continuous Integration
Step 1: Fork and clone our example repo and configure it to utilize a Contentful space
For this project we’ll be setting up a Continuous Integration pipeline for a barebones Flask App. The tutorial repository includes the Flask site and an export of a Contentful space which has a single content type.
After cloning our tutorial repository, create a new Contentful space and import the tutorial content model.
The Contentful part (required)
Create a new space using the
contentful-cli$ contentful space create --name "continuous delivery example"Note:* Creation of a space may result in additional charges if the free spaces available in your plan are exhausted.
Set the newly-created space as the default space for all further CLI operations. This will present a list of all available spaces – choose the one we just created
$ contentful space useImport the provided content model
(./import/export.json)into the newly-created space$ contentful space import --content-file ./import/export.json
These following steps to get the Flask site up and running are completely optional as it will only be used to test and build in our CI Pipeline. We won’t be making changes to the Python code outside of our tests in this example.
Local development environment (optional)
- Create a virtual environment
$ virtualenv env - Activate the virtual environment
$ source env/bin/activate - Install all Python dependencies
$ pip install -r requirements.txt - Start the Flask app
$ python myapp.py - Rename config file .env.example to .env and add missing keys
Step 2: Enable CircleCI to create a new Environment
Using CircleCI, we’ll create a list of steps that are run as part of the build, test and deployment phases. For this project, whenever we trigger a build phase, we want CircleCI to create a new environment, run a migration against that environment and then run our tests against that migrated environment. For each of those steps, we’ll need to write code that CircleCI can run to execute that step. Create a folder named scripts and inside it, a Javascript file called migrate.js.
To begin, we’ll need to import our dependencies and connect to Contentful via the content management SDK.
(async () => {
try {
const {promisify} = require('util');
const {readdir} = require('fs');
const readdirAsync = promisify(readdir);
const path = require('path');
const { createClient } = require('contentful-management');
const {default: runMigration} = require('contentful-migration/built/bin/cli');
// utility fns
const getVersionOfFile = (file) => file.replace('.js', '').replace(/_/g, '.');
const getFileOfVersion = (version) => version.replace(/\./g, '_') + '.js';
//
// Configuration variables
//
const [,, SPACE_ID, ENVIRONMENT_INPUT, CMA_ACCESS_TOKEN] = process.argv;
const MIGRATIONS_DIR = path.join('.', 'migrations');
const client = createClient({
accessToken: CMA_ACCESS_TOKEN
});
const space = await client.getSpace(SPACE_ID);
var ENVIRONMENT_ID = "";
let environment;
console.log('Running with the following configuration');
console.log(`SPACE_ID: ${SPACE_ID}`);Next we'll check if this script is running a feature branch on GitHub or if it's being merged into master. If it's being merged into master we'll add a UTC timestamp to the name of the environment we'll be creating.
if (ENVIRONMENT_INPUT == 'master'){
console.log(`Running on master.`);
ENVIRONMENT_ID = "master-".concat(getStringDate());
}else{
console.log('Running on feature branch');
ENVIRONMENT_ID = ENVIRONMENT_INPUT;
}
console.log(`ENVIRONMENT_ID: ${ENVIRONMENT_ID}`);Since enviroment names have strict requriements about allowed characters we'll need to have a function a that can format our timestamps for us. Given that multiple deployments to master can occur in a day, our timestamp will have granularity down to the minute.
function getStringDate(){
var d = new Date();
function pad(n){return n<10 ? '0'+n : n}
return d.toISOString().substring(0, 10)
+ '-'
+ pad(d.getUTCHours())
+ pad(d.getUTCMinutes())
}Next we'll check if an environment already exists and if it does, to destroy it. When we call this script, we’ll provide it with the name of the branch that it will use to create the new environment. Then, a new environment that’s a copy of master is created and polled until we received confirmation that it exists.
// ---------------------------------------------------------------------------
console.log(`Checking for existing versions of environment: ${ENVIRONMENT_ID}`);
try {
environment = await space.getEnvironment(ENVIRONMENT_ID);
if (ENVIRONMENT_ID != 'master'){
await environment.delete();
console.log('Environment deleted');
}
} catch(e) {
console.log('Environment not found');
}
// ---------------------------------------------------------------------------
if (ENVIRONMENT_ID != 'master'){
console.log(`Creating environment ${ENVIRONMENT_ID}`);
environment = await space.createEnvironmentWithId(ENVIRONMENT_ID, { name: ENVIRONMENT_ID });
}
// ---------------------------------------------------------------------------
const DELAY = 3000;
const MAX_NUMBER_OF_TRIES = 10;
let count = 0;
console.log('Waiting for environment processing...')
while (count < MAX_NUMBER_OF_TRIES) {
const status = (await space.getEnvironment(environment.sys.id)).sys.status.sys.id;
if (status === 'ready' || status === 'failed') {
if (status === 'ready') {
console.log(`Successfully processed new environment (${ENVIRONMENT_ID})`);
} else {
console.log('Environment creation failed');
}
break;
}
await new Promise(resolve => setTimeout(resolve, DELAY));
count++;
}Lastly we'll update our API keys to have access to the newly created environment so that our tests will be able to utilize it with the existing delivery API key we'll configure later in CircleCI.
// ---------------------------------------------------------------------------
console.log('Update API keys to allow access to new environment');
const newEnv = {
sys: {
type: 'Link',
linkType: 'Environment',
id: ENVIRONMENT_ID
}
}
const {items: keys} = await space.getApiKeys();
await Promise.all(keys.map(key => {
console.log(`Updating - ${key.sys.id}`);
key.environments.push(newEnv);
return key.update();
}));Step 3: Create a new Content Type named versionTracking
For CircleCI to know which migrations it should run, we’ll need to track which migrations have been run by adding a version number into Contentful. We accomplish this in Contentful by creating a new content model with an ID of versionTracking that has a single short-text-field named version.
You’ll also need to create one entry of your new content model with the value 1. We'll be using integers in this demo to track migrations.
Step 4: Create migrations
The Contentful Migration DSL allows us to:
- Create, delete and edit content models and content entries all in JavaScript.
- Establish a repeatable process of updating content model (where the Contentful web app does not)
- Track changes by integrating them into our version control; hence implementing CMS as Code.
The provided space in the example repo contains a list of Marvel superheroes; each having a name, GIF, first appearance and slug. For our first migration, we’ll modify the content model to include an author field.
Start by creating a new folder in the root directory named migration—this is where we’ll place all our migrations. We’ll need to create an empty migration file to represent the initial import that we did in step 1. Create 1.js and include the following code:
module.exports = function runMigration(migration) {
return;
};Next, create a JavaScript file named 2.js with the following code to add an author field to our content model:
module.exports = function runMigration(migration) {
const post = migration.editContentType("post");
post
.createField("author")
.name("author")
.type("Symbol")
.required(false);
};Lastly, we’ll create a second migration named 3.js to set the newly created author field of each hero as Stan Lee.
module.exports = function (migration) {
migration.transformEntries({
contentType: 'post',
from: ['author'],
to: ['author'],
transformEntryForLocale: function (fromFields, currentLocale) {
const author = "Stan Lee";
return { author: author };
}
});
};Since we’ve changed our content model, we’ll also need to update our tests. Update the test_content_type_post function in the test_app.py file with the following:
def test_content_type_post(self, contentful_client):
"""Test content model of a post"""
post_content_type = contentful_client.content_type("post")
# Expect 5 fields in a post now that we’ve added an author field.
assert len(post_content_type.fields) == 5
title = next(d for d in post_content_type.fields if d.id == "title")
assert title.id == "title"
assert title.type == "Symbol"And add the following test into our function:
author = next(d for d in post_content_type.fields if d.id == "author")
assert author.id == "author"
assert author.type == "Symbol"Step 5: Update migrate.js to run our migrations.
Now that we have a directory of migrations to run against our space, we’ll need to update our script so that CircleCI can run those migrations for us.
First we'll append to migrate.js code to set our default locale, check what migrations we have available and the finally detect what migrations need to be run by looking at our versionTracking entry.
// ---------------------------------------------------------------------------
console.log('Set default locale to new environment');
const defaultLocale = (await environment.getLocales()).items
.find(locale => locale.default).code;
// ---------------------------------------------------------------------------
console.log('Read all the available migrations from the file system');
const availableMigrations = (await readdirAsync(MIGRATIONS_DIR))
.filter(file => /^\d+?\.js$/.test(file))
.map(file => getVersionOfFile(file));
// ---------------------------------------------------------------------------
console.log('Figure out latest ran migration of the contentful space');
const {items: versions} = await environment.getEntries({
content_type: 'versionTracking'
});
if (!versions.length || versions.length > 1) {
throw new Error(
'There should only be one entry of type \'versionTracking\''
);
}
let storedVersionEntry = versions[0];
const currentVersionString = storedVersionEntry.fields.version[defaultLocale];
// ---------------------------------------------------------------------------
console.log('Evaluate which migrations to run');
const currentMigrationIndex = availableMigrations.indexOf(currentVersionString);
if (currentMigrationIndex === -1) {
throw new Error(
`Version ${currentVersionString} is not matching with any known migration`
);
}
const migrationsToRun = availableMigrations.slice(currentMigrationIndex + 1);
const migrationOptions = {
spaceId: SPACE_ID,
environmentId: ENVIRONMENT_ID,
accessToken: CMA_ACCESS_TOKEN,
yes: true
};Lastly we'll run all the migrations that we've evaluated as being necessary and update the version number.
// ---------------------------------------------------------------------------
console.log('Run migrations and update version entry');
while(migrationToRun = migrationsToRun.shift()) {
const filePath = path.join(__dirname, '..', 'migrations', getFileOfVersion(migrationToRun));
console.log(`Running ${filePath}`);
await runMigration(Object.assign(migrationOptions, {
filePath
}));
console.log(`${migrationToRun} succeeded`);
storedVersionEntry.fields.version[defaultLocale] = migrationToRun;
storedVersionEntry = await storedVersionEntry.update();
storedVersionEntry = await storedVersionEntry.publish();
console.log(`Updated version entry to ${migrationToRun}`);
}Step 6: Update migrate.js to update our master alias during deployment.
If this migration script was called as part of a deployment we'll have created a new environment named master- with at timestamp appended to the end. We'll need to add some code to update our environment alias with the ID of master to refer to this created enviroment.
// ---------------------------------------------------------------------------
console.log('Checking if we need to update master alias');
if (ENVIRONMENT_INPUT == 'master'){
console.log(`Running on master.`);
console.log(`Updating master alias.`);
await space.getEnvironmentAlias('master')
.then((alias) => {
alias.environment.sys.id = ENVIRONMENT_ID
return alias.update()
})
.then((alias) => console.log(`alias ${alias.sys.id} updated.`))
.catch(console.error);
console.log(`Master alias updated.`);
}else{
console.log('Running on feature branch');
console.log('No alias changes required');
}
console.log('All done!');
} catch(e) {
console.error(e);
process.exit(1);
}
})();The finalized migrate.js file should be identical to the version in our completed example.
Step 6: Create CircleCI Config File
The final step before we can push our code to GitHub is to create a CircleCI config file. We’ll start with the default CircleCI config file, which is included in the example directory, and make a few changes. As CircleCI instructs, save, the config.yml file is inside a folder named .circleci on the root directory.
We’ll need to add several modifications to the default config.yml provided by CircleCI.
Start by updating the image to include a version that comes with Node since we’ll need that to use the migration DSL:
- image: circleci/python:3.6.5-nodeUpdate the install dependencies step to additionally install the requirements listed in the package.json provided in the repo:
- run:
name: install dependencies
command: |
python3 -m venv venv
. venv/bin/activate
pip install -r requirements.txt
npm installAdd a step to prepare the environment for testing immediately after installing dependencies, but before running the tests. This step will utilize the previously created migrate.js file we created in steps 2.
- run:
name: Preparing environment for testing
command: |
. venv/bin/activate
scripts/migrate.js $SPACE_ID "CI_$CIRCLE_BRANCH" $MANAGEMENT_API_KEYUpdate the test step to utilize the new environment.
# run tests!
- run:
name: run tests
command: |
. venv/bin/activate
pytest --environment-id="CI_$CIRCLE_BRANCH"Once completed the config.yml file should look like this.
If we want to optionally include the deployment to Heroku in our initial CI rollout, we can do so utilizing the example deployment config and Heroku setup files that CircleCI provides for Flask deployments.
deploy:
docker:
- image: circleci/python:3.6.5-node-browsers
environment:
# Update HEROKU_APP with your application Name.
HEROKU_APP_NAME: "migration-env-demo"
steps:
- checkout
# Download and cache dependencies
- restore_cache:
keys:
- v1-dependencies-{{ checksum "package.json" }}
# fallback to using the latest cache if no exact match is found
- v1-dependencies-
- run:
name: install dependencies
command: |
npm install
- save_cache:
paths:
- node_modules
key: v1-dependencies-{{ checksum "package.json" }}
- run:
name: Creating new master environment and setting master alias
command: |
scripts/migrate.js $SPACE_ID master $MANAGEMENT_API_KEY
- run:
name: setup Heroku
command: bash .circleci/setup-heroku.sh
- deploy:
name: Deploy Master to Heroku
command: |
if [ "${CIRCLE_BRANCH}" == "master" ]; then
git push heroku master
heroku run python manage.py deploy
heroku restart
fiGiven that our tests could potentially take upto a few minutes, we'll be reruning our migration against a clean copy of master to pick up any content changes that might have occured during this process. In the event of any issues our previous copy of master will still exist and we can manually change the enviroment alias for master back to a previous version.
Step 7: Push code to GitHub and configure CircleCI
Now that we’ve made the changes needed for the project, we can push the code to GitHub on master. After logging in into the CircleCI website, click “Set Up Project” on the repository we’re using.
Click “Start building”—this first build will fail because it’ll be missing environment variables. From this screen, click the gear icon to be taken to the project settings. The project settings page allows us to access the environment variables where we can add the following keys: DELIVERY_API_KEY,
MANAGEMENT_API_KEY, SPACE_ID, HEROKU_LOGIN and HEROKU_API_KEY.
Rerun the build and everything should pass!
What’s next?
After following these steps, we’ll have a fully-functional CI pipeline integrated with Contentful. Any time someone make a PR, CircleCI will automatically run tests and create a new environment based on the branch name that will be used to verify code quality. If all our tests pass and a deploy occurs we'll create a new environment and update our environment alias for master to target that new environment.
Using this example, to run a migration against Contentful create a new migration file inside the migration folder and bump up the version number. With this method, if we need to revert, it's simple to do so by updating our enviroment allias for master to it's previous iteration. However, any bad migrations should not affect a production space since any failed tests will cause the build phase of this CI pipeline to not trigger. In fact, since CircleCI runs tests and migrations that are part of the PR review process, a bad migration should theoretically never hit production assuming it isn’t accidentally merged into master on GitHub.