Skip to content. | Skip to navigation

Personal tools

>>> ''.join(word[:3].lower() for word in 'David Isaac Glick'.split())

‘davisagli’

Navigation

You are here: Home

David Glick – Plone developer

by admin posted Apr 06, 2010 02:48 AM

In-browser integration testing with Windmill

by David Glick posted Jun 10, 2010 02:16 AM

Windmill makes it easy to run automated tests of a Plone project in a real browser.

I really like writing integration tests for web projects using the Zope testbrowser, which is a convenience API around the mechanize library. But its Achilles heel has always of course been that it only operates on the HTML response, so can't test interactive functionality built with Javascript and AJAX. So I've wanted to try one of the options for running tests in a real browser for a while. Actually it's a testament to the utility of the Zope testbrowser (or my own laziness?) that I made it this long. But the Plone resource customizer uses a lot of AJAX, so it was time.

So first I tried Selenium. I'd heard about Selenium from a number of people, and it's cool. There is a Firefox plugin that lets you record actions and assertions and play them back. It also lets you export these tests to Python to be run through the selenium Python bindings in conjunction with Selenium RC. There's even a Selenium Grid for running tests in parallel on multiple machines.

Unfortunately, while I found collective.ploneseltest which looked like just what I needed—it provides a base Selenium test case for use with the Zope testrunner—I had trouble getting it to actually work. Using version 1.0.3 of Selenium RC and of the Python bindings, Selenium RC was sending an extra HEAD request before each GET request, which was getting interpreted incorrectly by the ZPublisher during traversal in Zope.  A query to Twitter yielded the information (from the creator of selenium himself) that new Python bindings for selenium 2.0 are available as of last week, but I found they don't work (at least not yet) with Python 2.4. And anyway, by that point Martin Aspeli had also replied to my query and suggested trying Windmill instead.

Windmill is another web testing framework that actually seems to have quite similar functionality to Selenium, at least from a cursory examination. It also has an in-browser controller for recording, and can export tests to Python code. There is a Zope testrunner integration for Windmill too, in the niteoweb.windmill package.

And Windmill was quite a bit easier to get working. I just added the following to my package's setup.py to define a new installation "extra":

extras_require = {
    'test': ['niteoweb.windmill',],
},

And then modified my buildout's test runner to include that extra:

[test]
recipe = zc.recipe.testrunner
eggs =
    ${instance:eggs}
    plone.app.skineditor [test]
defaults = ['--exit-with-status', '--auto-color', '--auto-progress']

I added a new test module called test_integration.py and made it use the base test case from niteoweb.windmill:

import unittest
from niteoweb.windmill import WindmillTestCase
from Products.PloneTestCase.setup import setupPloneSite
from Products.PloneTestCase.layer import onsetup
from Products.Five.zcml import load_config
from Testing import ZopeTestCase as ztc

@onsetup
def load_zcml():
    import plone.app.skineditor
    load_config('configure.zcml', plone.app.skineditor)
    ztc.installPackage('plone.app.skineditor')

load_zcml()
setupPloneSite(products=['plone.app.skineditor'])

class SkinEditorIntegrationTestCase(WindmillTestCase):

    def afterSetUp(self):
        """Setup for each test
        """
        ztc.utils.setupCoreSessions(self.app)
        self.setRoles(['Manager'])
        self.login_user()

    def test_customize_logo(self):
        import pdb; pdb.set_trace()

def test_suite():
    return unittest.defaultTestLoader.loadTestsFromName(__name__)

(Most of this is standard test setup boilerplate. I could probably be a bit more sophisticated about the test setup and use layers or something, but this is the tried and true test setup I've been using since Martin published it in his book. The load_zcml method and setupPloneSite method are deferred and run when the Plone test layer is set up by the test runner.)

So far I just added one test that enters pdb. At this point, I can run the test with bin/test -s plone.app.skineditor, and Windmill will start up the ZServer with a dummy Plone site, fire up a browser and a controller window, and then pause at the pdb. (Unlike Selenium, I don't have to install a browser plugin or have another process running first. Nice!) Then I can play around with the controller and start recording tests.  The afterSetUp method runs before each test, and calls niteoweb.windmill's "login_user" helper, which does an initial login to the site as a Manager user.  It also tells the test infrastructure to support sessions, which are needed for my app.

Actually recording a test is mostly a point-and-click affair—hit the "record" button and go at it—but with a few caveats that I'll note below. It took a little bit for me to get used to the ways of selecting parts of the page and making assertions, but at least Windmill provides some flexibility. You can select by id, via XPath expressions, via JQuery selectors, or several other methods.  And assertions range from asserting that certain text is on the page to asserting that an arbitrary Javascript expression evaluates to true.

Here's the full test case I ended up with (dumped from the controller using the "save" button):

def test_customize_logo(self):
    client = self.wm
    client.click(id=u'user-name')
    # load customizer
    client.click(link=u'Site Setup')
    client.waits.forPageLoad(timeout=u'20000')
    client.click(link=u'Theme Editor')
    # go into advanced mode and customize based on the logo in the
    # non-active "plone_images" layer (Windmill doesn't do file uploads)
    client.click(link=u'Advanced')
    client.click(id=u'plone-app-skineditor-name-field')
    client.type(text=u'logo', id=u'plone-app-skineditor-name-field')
    client.click(id=u'plone-app-skineditor-filter-button')
    client.waits.forElement(timeout=u'', id=u'plone-app-skineditor-browser')
    client.click(xpath=u"//a[@id='skineditor-logo.png']/dt")
    client.waits.forElement(xpath=u"//dd[@class='plone-app-skineditor-layers']")
    client.click(jquery=u"('a[href*=plone_images/logo.png/manage_main]')[0]")
    client.waits.forElement(jquery=u"('#pb_1 input[value=Customize]')")
    client.click(name=u'submit')
    client.waits.forElement(timeout=u'', id=u'pb_2')
    # now reload and make sure the logo has the height we expect from the
    # customized image
    client.refresh()
    client.asserts.assertJS(js=u"$('#portal-logo').height() == 57")
    # now remove the customization
    client.click(id=u'plone-app-skineditor-name-field')
    client.type(text=u'logo', id=u'plone-app-skineditor-name-field')
    client.click(id=u'plone-app-skineditor-filter-button')
    client.waits.forElement(timeout=u'', id=u'skineditor-logo.png')
    client.click(xpath=u"//a[@id='skineditor-logo.png']/dt")
    client.waits.forElement(xpath=u"//dd[@class='plone-app-skineditor-layers']")
    client.click(link=u'Remove')
    # and confirm we're back to the original height
    client.refresh()
    client.asserts.assertJS(js=u"$('#portal-logo').height() == 56")

And here's what it looks like to run that. (No clicking on my part involved!) The excitement starts about 0:17...


Finally, here are a few caveats I ran into while setting up my first test, to customize the logo using plone.app.skineditor, and some last observations.

  • Windmill can't do file uploads. This is a limitation of browser Javascript support / sandboxing, not of Windmill per se.  It would be nice if there were some command that would prime the Windmill HTTP proxy to add a particular file to the next HTTP request that comes through, so that uploads could at least be faked.
  • Windmill could be a bit smarter about handling AJAX requests when recording.  I needed to manually add a "waitForElement" step after each click that resulted in an AJAX load, to make sure that the load was complete before subsequent steps run.  It would be nice if there was a "wait for all ajax to complete" command so that I didn't have to identify a particular element to wait for.
  • I wish that there was a wrapper that would let me control Windmill using the same API as zope.testbrowser, to make it easier to convert existing tests to full browser tests.
  • So far I've only tried running tests in my default browser (Firefox), but it's possible to run in other browsers as well.  The niteoweb.windmill page on PyPI gives an example of a test layer for doing this.
  • It remains to be seen how cumbersome this sort of test will be to keep up-to-date as the product evolves.

Overall Windmill provides a decent experience for recording tests and a really nice one for running them. Perhaps it is a tool we can use to do real browser testing for Plone core?

If you've used Windmill and have any insights into how to use it effectively, I'd love to hear your thoughts in the comments.

3 comments

Review of "Plone 3 Products Development Cookbook"

by David Glick posted Jun 06, 2010 09:24 PM

Packt Publishing recently released their new title, "Plone 3 Products Development Cookbook"

Last month Packt Publishing released a new title to add their growing line of books on Plone development — Plone 3 Products Development Cookbook by Juan Pablo Giménez and Marcos F. Romero, reviewed by Martin Aspeli, Alec Mitchell, and Emanuel Sartor. The book aims to be a useful recipe-oriented resource for beginner- to intermediate-level developers of Plone add-on products. Having read the review copy Packt was kind enough to send me, I can recommend it as an up-to-date and more example-oriented complement to Packt's still-great established title on Plone 3 development, Martin Aspeli's Professional Plone Development.

I am a bit skeptical of the traditional cookbook format, in which a series of disjointed "recipes" are presented for dealing with various scenarios.  In practice these are most useful when your needs match the problem expressed in the recipe exactly, which is seldom. So I was glad to see that the authors of this cookbook have departed from that pattern somewhat. The book does contain a series of recipes, but they are all related to an overall project, to create a digital newspaper website, which runs throughout the book. This helps provide continuity and give clarity on which pieces of the included recipes are needed for the particular project as opposed to being general.

The "recipe" concept is still evident in the book's format, though. Each chapter is divided into several tasks, each of which has 2 main parts: a "How to do it" section which lists the steps needed to complete the task, followed by a "How it works" section which describes the theory behind the task as well as the finer points of any of the steps which were non-obvious. It may seem a little weird that the explanation is separate from the instructions, but I think it could be a useful approach for a reader who learns well by example, and wants to try to figure out the meaning of the steps as they go before the answers are given. This format is one of the main things that sets this book apart from Professional Plone Development, which otherwise covers some similar topics.  I would recommend the latter to someone who learns best by reading first or who wants a reference, and this book to someone who learns best by doing first.

In addition, this book is somewhat more up-to-date with current tools and techniques (though I anticipate that will be fully rectified in the upcoming Plone 4 edition of PPD later this year).  Though the Products Development Cookbook goes into less depth in terms of explanation, the authors have been quite good about including links to up-to-date online resources such as the Plone Developer Manual where they don't have space to cover a subject in depth. The title, Plone 3 Products Development Cookbook, is a bit of a misnomer, by the way.  As far as I know, almost all the techniques described should work just as well in Plone 4.  (Unfortunately this repeats a mistake in naming from Erik Rose's Plone 3 for Education and Alex Clark's Plone 3 Site Administration, which also both aim to be relevant for Plone 4...it is a shame Packt didn't realize it is possible to write a book that targets both Plone 3 and Plone 4.)

I particularly enjoyed the chapters discussing content type creation. The book has examples of building an Archetype using ArgoUML and ArchGenXML, of building an Archetype using the 'archetype' ZopeSkel template, of creating a basic type using plone.app.content, and of creating a type using Dexterity...so this makes for an interesting quick comparison of the steps required in each case. (It is not an in-depth discussion of any of these approaches, though.  I would use the online manuals for Archetypes and Dexterity to learn the finer points of either system.)

There are also several chapters that did a good job of recording some of the lore about Plone development tools and processes that has mostly been passed along in blogs and sprints so far.  For example, it covers the use of IPython and ipdb, plone.reload, in-browser testing with selenium, load testing with funkload, and a full production buildout including ZEO, varnish, pound, supervisor, and assorted utilities.  As a seasoned Plone developer myself, I even learned a few things from these sections. In addition to the topics mentioned above, the book also covers installing Plone, testing, internationalization, workflow and permissions, KSS, portlets, use of the Zope Component Architecture, and integrating with an external system (OpenX) via XML-RPC.

For more information, orders, and errata, visit Packt Publishing's page for Plone 3 Products Development Cookbook.  As usual for Packt, the book is available both in print and as an ebook.

Introducing the Plone resource customizer

by David Glick posted Jun 02, 2010 01:50 AM

Thanks to the sprint at Plone Symposium East, the Plone resource customizer is almost ready for a beta release.

Going into the ZMI to edit templates and stylesheets and replace images sucks. Especially once we made it so there are two places you might need to go in Plone 3 (portal_skins and portal_view_customizations).  I've been working on a better way, the (drumroll please) Plone resource customizer*.

Thanks to the great work of my fellow sprinters at Plone Symposium East, I'm now ready to give a preview of the tool, which hopefully will see a first beta release real soon now™. Here's the screencast...

I forgot to mention in the screencast that Eric Steele has also started work on integrating this tool with Gloworm, so that it will be possible to find a resource to customize just by pointing and clicking (a la Firebug).

Currently the customizer displays items from CMF skin layers, browser view templates, viewlets, and portlets. Support for other things like browser resources or ZMI pages could probably be added, at least in a read-only fashion. The infrastructure is flexible enough to support new ways of registering resources that haven't been invented yet.

For now, if you're adventurous and want to try out the customizer, get a copy of the Plone 4 coredev buildout, and run it using -c experimental/skineditor.cfg

We are tracking bugs and ideas for improvement in the Plone bug tracker, with the component set to "Skin Editor."

This is only the beginning...David Bain is working on the customizer as part of the Google Summer of Code, and we've got lots of ideas about how to make it even more useful.

*better names hereby solicited :)

7 comments
David Glick

David Glick

I am a problem solver trying to make websites easier to build.

Currently I do this in my spare time as a member of the Plone core team, and during the day as an independent web developer specializing in Plone and custom Python web applications.