Using PhantomCSS for Regression Testing Your CSS

I finally had a chance to play with PhantomCSS, a CSS regression testing tool that compares screenshots to see if CSS changes have affected the design, and wanted to share my experience with you. We are using it at Ariba to validate that I do not break any designs as I migrate us from two giant, unmanageable CSS files into smaller, modularize and better written lessCSS files.

PhantomCSS uses CasperJS, built on PhantomJS as a testing framework with screenshot capabilities, and ResembleJS to compare images.

It was surprisingly easy to get it to work when I used all the sub-components packaged with the download, but incredible difficult when I tried to use the sub-components I already had installed via npm.

Getting ready

The easiest way to get started is to clone the project and cd into it:

git clone https://github.com/Huddle/PhantomCSS.git
cd PhantomCSS

The project contains a version of CasperJS, resembleJS, and PhantomJS. However, the included PhantomJS only works on windows. For installation on other systems, check out the PhantomJS Installation Guide (version >=1.2 was required at the time of this writing). Except for PhantomJS, you should make sure you are using the included software versions (if you don’t change anything, then you should be using the included packages). I initially tried to use the CasperJS that I had previously installed from NPM and wasted a lot of time trying to get it PhantomCSS working with the latest version of CasperJS, when the included package works right away.

How do it…

To ensure everything is working run the test applications twice:

phantomjs demo/testsuite.js

You should see the following the first time:

Must be your first time?
Some screenshots have been generated in the directory ./screenshots
This is your 'baseline', check the images manually. If they're wrong, delete the images.
The next time you run these tests, new screenshots will be taken.  These screenshots will be compared to the original.
If they are different, PhantomCSS will report a failure.

THE END.

And this the second:

PhantomCSS found: 4 tests.
None of them failed. Which is good right?
If you want to make them fail, go change some CSS - weirdo.

The test application creates a simple nodeJS webserver and sends a couple of requests against it. The screenshots will be in the screenshots directory. If you want to see what happens when a regression is detected, change a style or visible text in demo/coffeemachine.html and rerun testsuite.js.

Now, to get started with your own test. Create a new testing file:

touch demo/mytest.js

And put the following code in the file (trimmed from testsuite.js):

/*
	Initialise CasperJs
*/

phantom.casperPath = 'CasperJs';
phantom.injectJs(phantom.casperPath + '/bin/bootstrap.js');
phantom.injectJs('jquery.js');

var casper = require('casper').create({
	viewportSize: {
		width: 1027,
		height: 800
	}
});

/*
	Require and initialise PhantomCSS module
*/

var phantomcss = require('./phantomcss.js');

phantomcss.init({
	screenshotRoot: './screenshots',
	failedComparisonsRoot: './failures'
});

/*
	The test scenario
*/

var url = 'http://www.mattsnider.com'; // replace with your URL

casper.
	start(url).
	// screenshot the initial page load
	then(function() {
		phantomcss.screenshot('<SELECTOR_TO_SCREENSHOT>', '<LABEL_SCREENSHOT>');
	}).
	then(function() {
		// do something
	}).
	// second screenshot
	then(function() {
		phantomcss.screenshot('<SELECTOR_TO_SCREENSHOT>', '<LABEL_SCREENSHOT>');
	});

/*
	End tests and compare screenshots
*/

casper.
	then(function now_check_the_screenshots() {
		phantomcss.compareAll();
	}).
	run(function end_it() {
		console.log('\nTHE END.');
		phantom.exit(phantomcss.getExitStatus());
	});

Change the casper test as necessary and run it against your site:

phantomjs demo/mytest.js

How it works…

Most of the navigation and testing work is handled by CasperJS, so most likely any you want to do/test is documented here. That said, at the time of this writing PhantomCSS was still on Casper 1.0.2, while the documentation was for version 1.1.0 (I couldn’t find an active version of 1.0.2 documentation, if you find one, please let me know in the comments). The simple example we build (mytest.js) just opens a URL, takes a screenshot, does something that you specify, and takes another screenshot. From this starting point you should be able to begin building more comprehensive tests.

The first time PhantomCSS runs it will perform all specified operations and take a series of screenshots (as specified by calls to phantomcss.screenshot('', '');), but the phantomcss.compareAll(); will print out a message saying that there is nothing to compare. You have just captured screenshots of the unchanged, baseline version of your site. Now go make changes to the CSS of your site and next time PhantomCSS runs, it will create new screenshots and compare them to the original. Should the screenshots be different, then PhantomCSS will error and tell you what steps failed.

The boilerplate at the top of mytest.js is used to find and hook all the component libraries into PhantomJS, before we create a webpage using CasperJS. Next PhantomCSS is initialized, specifying the screenshot and failure directories. There are a lot of other configuration properties that may be specified, but these two are the most important for simple tests. CasperJS is invoked by calling .start(url). CasperJS uses the promise pattern, so you can chain function calls (a shown in the example above), or you can in call casper. individually (as shown by the end section).

There’s more…

Lastly, here are some pointers and gotchas with CasperJS that might help you:

  • Use this.echo("your string"); from inside of a callback function to print something to the terminal (or you may also use require("utils").dump("your string");), but console.log won’t print to the terminal.
  • Use this.echo(this.getHTML(selector, true)) when you aren’t sure that the correct element was found by the selector.
  • Inside the this.evaluate(callback) and this.thenEvaluate(callback) callback functions, you can run JavaScript against the page as you normally would (such as document.getElementById(id)), but you won’t have access to the CasperJS helper functions.
  • Attach listeners to CasperJS if you want to print the messages and alerts that are triggered by the HTML page you are evaluating (see code below).

function printMsg(msg) {
    casper.echo(msg);
}
casper.on("remote.alert", printMsg);
casper.on("remote.message", printMsg);