Skip to content. | Skip to navigation

Personal tools

>>> ''.join(word[:3].lower() for word in 'David Isaac Glick'.split())

‘davisagli’

Navigation

You are here: Home / Blog / In-browser integration testing with Windmill

In-browser integration testing with Windmill

by David Glick posted Jun 09, 2010 11:16 PM
Windmill makes it easy to run automated tests of a Plone project in a real browser.

I really like writing integration tests for web projects using the Zope testbrowser, which is a convenience API around the mechanize library. But its Achilles heel has always of course been that it only operates on the HTML response, so can't test interactive functionality built with Javascript and AJAX. So I've wanted to try one of the options for running tests in a real browser for a while. Actually it's a testament to the utility of the Zope testbrowser (or my own laziness?) that I made it this long. But the Plone resource customizer uses a lot of AJAX, so it was time.

So first I tried Selenium. I'd heard about Selenium from a number of people, and it's cool. There is a Firefox plugin that lets you record actions and assertions and play them back. It also lets you export these tests to Python to be run through the selenium Python bindings in conjunction with Selenium RC. There's even a Selenium Grid for running tests in parallel on multiple machines.

Unfortunately, while I found collective.ploneseltest which looked like just what I needed—it provides a base Selenium test case for use with the Zope testrunner—I had trouble getting it to actually work. Using version 1.0.3 of Selenium RC and of the Python bindings, Selenium RC was sending an extra HEAD request before each GET request, which was getting interpreted incorrectly by the ZPublisher during traversal in Zope.  A query to Twitter yielded the information (from the creator of selenium himself) that new Python bindings for selenium 2.0 are available as of last week, but I found they don't work (at least not yet) with Python 2.4. And anyway, by that point Martin Aspeli had also replied to my query and suggested trying Windmill instead.

Windmill is another web testing framework that actually seems to have quite similar functionality to Selenium, at least from a cursory examination. It also has an in-browser controller for recording, and can export tests to Python code. There is a Zope testrunner integration for Windmill too, in the niteoweb.windmill package.

And Windmill was quite a bit easier to get working. I just added the following to my package's setup.py to define a new installation "extra":

extras_require = {
    'test': ['niteoweb.windmill',],
},

And then modified my buildout's test runner to include that extra:

[test]
recipe = zc.recipe.testrunner
eggs =
    ${instance:eggs}
    plone.app.skineditor [test]
defaults = ['--exit-with-status', '--auto-color', '--auto-progress']

I added a new test module called test_integration.py and made it use the base test case from niteoweb.windmill:

import unittest
from niteoweb.windmill import WindmillTestCase
from Products.PloneTestCase.setup import setupPloneSite
from Products.PloneTestCase.layer import onsetup
from Products.Five.zcml import load_config
from Testing import ZopeTestCase as ztc

@onsetup
def load_zcml():
    import plone.app.skineditor
    load_config('configure.zcml', plone.app.skineditor)
    ztc.installPackage('plone.app.skineditor')

load_zcml()
setupPloneSite(products=['plone.app.skineditor'])

class SkinEditorIntegrationTestCase(WindmillTestCase):

    def afterSetUp(self):
        """Setup for each test
        """
        ztc.utils.setupCoreSessions(self.app)
        self.setRoles(['Manager'])
        self.login_user()

    def test_customize_logo(self):
        import pdb; pdb.set_trace()

def test_suite():
    return unittest.defaultTestLoader.loadTestsFromName(__name__)

(Most of this is standard test setup boilerplate. I could probably be a bit more sophisticated about the test setup and use layers or something, but this is the tried and true test setup I've been using since Martin published it in his book. The load_zcml method and setupPloneSite method are deferred and run when the Plone test layer is set up by the test runner.)

So far I just added one test that enters pdb. At this point, I can run the test with bin/test -s plone.app.skineditor, and Windmill will start up the ZServer with a dummy Plone site, fire up a browser and a controller window, and then pause at the pdb. (Unlike Selenium, I don't have to install a browser plugin or have another process running first. Nice!) Then I can play around with the controller and start recording tests.  The afterSetUp method runs before each test, and calls niteoweb.windmill's "login_user" helper, which does an initial login to the site as a Manager user.  It also tells the test infrastructure to support sessions, which are needed for my app.

Actually recording a test is mostly a point-and-click affair—hit the "record" button and go at it—but with a few caveats that I'll note below. It took a little bit for me to get used to the ways of selecting parts of the page and making assertions, but at least Windmill provides some flexibility. You can select by id, via XPath expressions, via JQuery selectors, or several other methods.  And assertions range from asserting that certain text is on the page to asserting that an arbitrary Javascript expression evaluates to true.

Here's the full test case I ended up with (dumped from the controller using the "save" button):

def test_customize_logo(self):
    client = self.wm
    client.click(id=u'user-name')
    # load customizer
    client.click(link=u'Site Setup')
    client.waits.forPageLoad(timeout=u'20000')
    client.click(link=u'Theme Editor')
    # go into advanced mode and customize based on the logo in the
    # non-active "plone_images" layer (Windmill doesn't do file uploads)
    client.click(link=u'Advanced')
    client.click(id=u'plone-app-skineditor-name-field')
    client.type(text=u'logo', id=u'plone-app-skineditor-name-field')
    client.click(id=u'plone-app-skineditor-filter-button')
    client.waits.forElement(timeout=u'', id=u'plone-app-skineditor-browser')
    client.click(xpath=u"//a[@id='skineditor-logo.png']/dt")
    client.waits.forElement(xpath=u"//dd[@class='plone-app-skineditor-layers']")
    client.click(jquery=u"('a[href*=plone_images/logo.png/manage_main]')[0]")
    client.waits.forElement(jquery=u"('#pb_1 input[value=Customize]')")
    client.click(name=u'submit')
    client.waits.forElement(timeout=u'', id=u'pb_2')
    # now reload and make sure the logo has the height we expect from the
    # customized image
    client.refresh()
    client.asserts.assertJS(js=u"$('#portal-logo').height() == 57")
    # now remove the customization
    client.click(id=u'plone-app-skineditor-name-field')
    client.type(text=u'logo', id=u'plone-app-skineditor-name-field')
    client.click(id=u'plone-app-skineditor-filter-button')
    client.waits.forElement(timeout=u'', id=u'skineditor-logo.png')
    client.click(xpath=u"//a[@id='skineditor-logo.png']/dt")
    client.waits.forElement(xpath=u"//dd[@class='plone-app-skineditor-layers']")
    client.click(link=u'Remove')
    # and confirm we're back to the original height
    client.refresh()
    client.asserts.assertJS(js=u"$('#portal-logo').height() == 56")

And here's what it looks like to run that. (No clicking on my part involved!) The excitement starts about 0:17...


Finally, here are a few caveats I ran into while setting up my first test, to customize the logo using plone.app.skineditor, and some last observations.

  • Windmill can't do file uploads. This is a limitation of browser Javascript support / sandboxing, not of Windmill per se.  It would be nice if there were some command that would prime the Windmill HTTP proxy to add a particular file to the next HTTP request that comes through, so that uploads could at least be faked.
  • Windmill could be a bit smarter about handling AJAX requests when recording.  I needed to manually add a "waitForElement" step after each click that resulted in an AJAX load, to make sure that the load was complete before subsequent steps run.  It would be nice if there was a "wait for all ajax to complete" command so that I didn't have to identify a particular element to wait for.
  • I wish that there was a wrapper that would let me control Windmill using the same API as zope.testbrowser, to make it easier to convert existing tests to full browser tests.
  • So far I've only tried running tests in my default browser (Firefox), but it's possible to run in other browsers as well.  The niteoweb.windmill page on PyPI gives an example of a test layer for doing this.
  • It remains to be seen how cumbersome this sort of test will be to keep up-to-date as the product evolves.

Overall Windmill provides a decent experience for recording tests and a really nice one for running them. Perhaps it is a tool we can use to do real browser testing for Plone core?

If you've used Windmill and have any insights into how to use it effectively, I'd love to hear your thoughts in the comments.

Dylan Jay says:
Jun 09, 2010 04:33 PM
This code although a little old, seemed to work for me. It used a mozilla plugin.
It would be nice to use selenium or windmill as additional backends for testbrowser.
I never got testbrowser recorder to work which is a pity. It would be nice to have a converter from windmill or selenium format to testbrowser.
David Ray says:
Jun 30, 2010 06:40 AM
David:

Great article. Very timely for us, as we were having issues with Selenium functioning in a consistent manner.

I figured out a workaround for mocking a file upload and have written about it here:

http://enrage-timer.com/[…]/
Jason Mehring says:
Jul 02, 2010 10:11 AM
I have also run into the the problem of file uploads in the past and was able to patch mozilla xulrunner to ignore security on file uploads. Its a bad thing to do :) And you would want to be sure you never used that version of mozilla on a public web site due to the self inflicted security hole!

I can post the patch if anyone is interested, but I would think a Mozilla recipe would need to be created for buildout to compile it.

I also recall an easier method was an ability to declare a security exception in mozilla startup config file. When a user loads the page they are asked to confirm the exception, which should not be a problem to do in windmill.

Following is a copy of the Mozilla prefs.js I used to override security (do not need to patch source code with this option, but you need to add something to javascript):

# Mozilla User Preferences

/* Do not edit this file.
 *
 * If you make changes to this file while the application is running,
 * the changes will be overwritten when the application exits.
 *
 * To make a manual change to preferences, you can visit the URL about:config
 * For more information, see http://www.mozilla.org/unix/customizing.html#prefs
 */

// Override the default user-agent string:
//user_pref("general.useragent.override", "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.0.0; hi, Mom) Gecko/20020604");
user_pref("general.useragent.override", "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.7) Gecko/20100106 Ubuntu/9.10 (karmic) Firefox/3.5.7");

// Enable the dump() global function. Prints to stdout.
user_pref("browser.dom.window.dump.enabled", true);

// Allow popups
user_pref("dom.disable_open_during_load", false);

// Disable slow-script alerts
user_pref("dom.max_script_run_time", 0);

// Turn off TLS warnings
user_pref("security.warn_submit_insecure", false);

// Turn off default browser check
user_pref("browser.shell.checkDefaultBrowser", false);

// Turn off quit warnings
user_pref("browser.warnOnQuit", false);

// Allow the hosts below to execute privileged script without warnings
user_pref("signed.applets.codebase_principal_support", true);

// Allow localhost to execute privileged scripts
//user_pref("capability.principal.codebase.p0.granted", "UniversalXPConnect UniversalBrowserRead UniversalBrowserWrite UniversalPreferencesRead UniversalPreferencesWrite UniversalFileRead");
user_pref("capability.principal.codebase.p0.id", "file:///home/jason/test.html");
//user_pref("capability.principal.codebase.p0.id", "file:///*");
//user_pref("capability.principal.codebase.p0.subjectName", "");

Add this to javascript to allow warning popup exception:

netscape.security.PrivilegeManager.enablePrivilege('UniversalFileRead');

I don't know what exists for browsers not based on Mozilla, so it may not be too useful if needing to test out other browsers.

ttyl,

Jason
Navigation