• Attending the Sea Sprint and pushing forward the Deco layout system
• Continuing to improve Dexterity's capabilities for building content types through the web
• Helping prepare the release of Plone 4.3 (including completing the demise of KSS)

I'll be doing this work on a volunteer basis, but if you've benefited from my contributions to Plone and/or want to help make sure I continue to have time available to work on it, please consider leaving me a tip:

Following the Plone conference, I am scheduling projects for mid-October and beyond, and am especially interested in:

• Chances to work with Plone, Pyramid, and/or other Python-based web frameworks—or train other developers on how to use them well
• Web projects that are interactive rather than merely presenting information
• Projects that expand what is possible for non-developers to accomplish using Plone
• Projects that are driven in part by a "social good" mission rather than merely profit

If you have something like that I can assist with, please be in touch!

Tasks completed

Ross Patterson wrote a tutorial introducing the basics of zc.buildout. He also started investigating what it would take to let Plone be installed by Microsoft's free Web Platform Installer for Windows/IIS.

Luke Brannon and Ross wrote some CSS for reStructuredText "admonishments" (i.e. note/warning/etc. boxes) that will be added to plone.org for when reST documents are rendered. Luke also refactored and tested Windows installation instructions in the knowledgebase, reviewed KB articles for accurate version metadata, and started looking into sorting KB topic pages by Plone version in addition to modified date.

Spanky Kapanka learned how to contribute to the collective developer manual, then worked on writing several quick start guides to walk Plone beginners through specific tasks. He completed one on subclassing a content type and started one on creating a custom form.

Bill Deegan edited the introduction to the collective developer manual for clarity and to present Plone's capabilities in a more positive light alongside the fair warnings of complexity. He also renamed the files in the collective docs to have a .rst extension so they render when viewed on github, and tested Spanky's quick start.

Mike Cullerton learned git and reStructuredText, then worked on a "quick start" guide on customizing views and viewlets.

Liz Leddy worked on a new set of documentation of how to do Plone core development, which will live in the coredev buildout. She also did a lot of cleanup in Plone's trac and moved information to buildout.coredev and to a new, streamlined "Get Involved" landing page which replaces the front page of trac as an entry point to participating in the Plone community.

Steve McMahon worked on writing a Plone sysadmin manual discussing how to install and configure Plone for deployment scenarios on various platforms. I think he is about two-thirds done and has not published the document yet.

Tyler Randles (of ploneconf panda head fame) and I worked on improving the template for the knowledgebase documentation. We added a box at the top which shows a color-coded warning about the document's version compatibility, and a box at the bottom with a brief invitation to contribute to editing the knowledgebase (you too can help review KB articles to make sure their metadata is correct!). We also created mr.crabby, your blue crabby friend for Plone—can you find him on plone.org? (Hint: search plone.org for info on the Cioppino Sprint.) Tyler also worked on illustrations for Steve's manual.

I, David Glick, also added a version selector to the search forms in the plone.org documentation section, fixed a bug in the knowledgebase permissions so that articles are actually editable by the right people (i.e. anyone logged in), reviewed some of Liz's core dev docs, added i18n support to plone.supermodel and plone.registry's GenericSetup import handler in order to fix a Plone 4.2 blocker, and did some planning about how to better organize Dexterity's documentation.

Fulvio Casali reviewed and tried out the existing Dexterity documentation and created a detailed list of suggested improvements which will be very helpful.

Florian Friesdorf wrote instructions for setting up a Plone development environment using nix, and added support for paged searches to pas.plugins.ldap.

Eric Steele worked on getting Plone 4.2 ready for final release, including a lot of work on the new-style collections. (It needed some work on the date criteria for feature parity with the existing collections, and some work to make sure that Plone's default news and event collections are the new style.) By the end of the sprint Eric announced that there are no more blockers for the second release candidate!

Hanno Schlichting joined the #sprint IRC channel remotely and continued his quest for an improved ZCatalog. He suggested some catalog optimizations for plone.org which I will hopefully review soon, and implemented two long-desired new features: "not" queries and sorting results by multiple indexes.

Lola and Zoey, the dogs, sprinted along the beach and made themselves generally lovable.

Highlights

A highlight of the sprint for many of us was the food. Mike made homemade pizzas and introduced us to whoopie pies with habanero. Spanky prepared breakfast after wonderful breakfast, including his recently perfected Hollandaise for eggs benedict with crab cakes. For snacks we had Luke's guacamole and Spanky and Tyler's invention of tater tot-bacon-pepperjack cheese bites. I baked sourdough bread to go with Steve's cioppino featuring fresh crab, mussels and clams, and Liz made a great chili that was dubbed "Holy Crap!" Some of these recipes have been posted to the sprint mailing list so look for them there.

Another highlight for me was participating in my first "code dojo." This is a technique for sharing knowledge where a group gathers to complete a task surrounding a single big TV or projector screen. One person gets to control the keyboard, another person tells the typist what to do, and everyone else can be consulted for advice. We used this technique to work toward solving issue 12796, as an exercise in how to fix a bug in Plone core for those who hadn't done so before (and as a "functional test" of Liz's new documentation of this process). We didn't have time to complete the task, but I think everyone had fun and learned something new (within the first five minutes I learned a new bash keyboard shortcut!).

One thing that did not happen at the sprint was a clear designation of a revitalized documentation team to make sure that our documentation is well-managed on an ongoing basis. Personally I feel that this is a role that is lacking in the community—Mikko and others are doing a fantastic job of getting people to add and update documentation in the collective developer manual, but I there is a need for a more focused manual and introductory documentation for people who are trying to learn Plone development for the first time rather than looking up particular tasks or topics—communally edited documentation is inevitably of varying quality and relevance, and newbies have no way to judge that. There is also a need to make sure that old documents are updated or marked as obsolete as appropriate. It's not entirely surprising that we've lacked this editing, as it is a thankless task (everyone can get excited about new documentation; someone always hates you when you delete something). I don't have answers but I hope we can brainstorm as a community about how to solve this problem (or maybe you can convince me I've diagnosed the wrong problem.)

But despite a lack of resolution of those questions, the sprint was a success in the realms of both fun and productivity. Thanks to the participants for a fantastic experience, to the Plone Foundation for sponsoring our lodging, and I look forward to seeing you at Cioppino 2013 if not before!

p.s. A special thanks to Spanky who provided a limo pickup and lodging to Fulvio and me after our train was canceled and we had to change travel plans at the last minute!

More photos

• Bill's
• mine

Plone does great at in-place editing: navigate to the thing you want to edit, then click the button and edit it. However, this paradigm breaks apart as soon as there is a need for a page to have multiple editable areas—such as for a homepage or section landing page.

At Groundwire, we used to deal with this problem by creating an ongoing series of very similar hacky one-off templates: the sort of template that would have have several areas which each pulled in the content from some item in a hidden folder of page components. Unfortunately this approach did not scale very well: it was tedious for us to set up new templates, and it was cumbersome for editors to remember how everything was set up in order to successfully make changes.

Last year we worked on the Net Impact website which has a different multi-part layout for each section landing page, and we realized that we needed to come up with a better solution. The requirements:

• Someone writing a template should be able to define an editable area of that template very simply, by just adding a line to the template that specifies the name of the area.
• There should be support for different types of editable areas; each type may have different settings when editing the area.
• Editing an area should be triggered by a pencil icon that shows up while hovering over the area for users who have permission to edit the area,
• All this should be done in a way that is simple to reuse for new sites.

Tiles to the rescue

We realized right away that our requirements were very similar to the functionality provided by the Deco project's implementation of "tiles." Deco is an ambitious project to provide drag-and-drop layout capabilities within Plone. Deco as a whole was not mature enough for us to feel comfortable using it, but I knew that the tile rendering was one of the older and more mature parts of Deco, and we realized that it would not take a lot of effort to use tile rendering without the rest of Deco.

A tile is a snippet that can be inserted into a template as a div with a data-tile attribute, like this:

<div data-tile="/Plone/@@mytile" />

Then some machinery in the publisher provided by plone.app.blocks performs the following steps:

1. It finds all the divs with a data-tile attribute (let's call them tile placeholders).
2. For each one, it performs a subrequest to fetch the contents of the tile. Using a URI makes tiles very flexible: a tile could be a browser view, or it could come from some external system.
3. The tile placeholder is replaced with the contents of the tile's body tag. If the tile has a head tag, its contents will be appended to the head of the page that includes the tile.

That's a great start! As it turns out, other parts of the tiles implementation also help support our use case:

• plone.tiles has a tile implementation which supports having multiple tile types. A tile turns out to basically be a browser view that also happens to have some associated data. (This is a lot like a portlet renderer, but one that can be added anywhere with a line in a template rather than needing to mess around with portlet managers.) Each type of tile can specify a different schema for its data, and that data can be persisted in different ways.
• plone.app.tiles provides an edit form that takes care of editing the data for a particular instance of a tile.

In practice: adding a rich text tile

So let's see how this plays out in practice. We are going to:

1. Set up the basic tile rendering machinery.
2. Implement a rich text tile that can be added anywhere, and that stores its contents in an annotation of the context where it is added.
3. Make sure that editors see a pencil icon that brings up an modal overlay to edit the tile.

The basics

Okay, let's get the basics set up.

1. Create a package that declares dependencies on: lxml, plone.app.blocks, plone.app.textfield, plone.app.tiles, and plone.tiles.
2. At the time of this writing, you need trunk checkouts of plone.app.blocks, plone.app.tiles, and plone.tiles.
3. Make sure that your configure.zcml includes <includeDependencies package="."/>.
4. Make sure the metadata.xml in your package's GenericSetup profile runs the default profiles from plone.app.blocks and plone.app.tiles as dependencies.
5. Install your package.

The tile

Add a tile.py with the following:

from zope.interface import Interface

from plone import tiles
from zope.schema import Text
from plone.app.textfield import RichText
from plone.app.textfield.interfaces import ITransformer


class IRichTextTileData(Interface):

    text = RichText(title=u'Text')


class RichTextTile(tiles.PersistentTile):

    def __call__(self):
        text = ''
        if self.data['text']:
            transformer = ITransformer(self.context, None)
            if transformer is not None:
                text = transformer(self.data['text'], 'text/x-html-safe')
        return '<html><body>%s</body></html>' % text

In configure.zcml, add:

  <plone:tile
      name="groundwire.tiles.richtext"
      title="Groundwire rich text tile"
      description="A tile containing rich text"
      add_permission="cmf.ModifyPortalContent"
      schema=".tile.IRichTextTileData"
      class=".tile.RichTextTile"
      permission="zope2.View"
      for="*"
      />

This defines a new tile type, called groundwire.tiles.richtext. This tile type has a schema with a single rich text field, and when it is rendered the tile will run the configured text through the safe HTML transform to make sure it is safe.

Wiring in the edit form

Now we just need to make sure that editors will have a way to access the edit interface for tiles.

Add the following javascript. Make sure you put a condition on it like "python:object.portal_membership.checkPermission('Modify portal content', object)" so that it will only run and add the edit links for users who have permission to edit.

jQuery(function($) {
  $('div[data-tile]').each(function() {
      $(this).addClass('tile-editable');
      var href = $(this).attr('data-tile');
      var edithref = href.replace(/@@/, '@@edit-tile/');
      $('<a class="tile-edit-link" href="' + edithref + '"><img height="16" src="pencil_icon.png" width="16" />')
        .appendTo($(this))
        .prepOverlay({
            subtype: 'iframe',
            config: {
                onClose: function() { location.reload(); }
            }
        });
  });
  
  // Check if tiledata is available and valid
  if (typeof(tiledata) !== 'undefined') {

      // Check action
      if (tiledata.action === 'cancel' || tiledata.action === 'save') {
          // Close dialog
          window.parent.jQuery('.link-overlay').each(function() {
              try {
                  window.parent.jQuery(this).overlay({api: true}).close();
              } catch(e) { }
          });
      }
  }
  
});

This adds an edit link to all the divs that have data-tile attributes. It also handles the "tiledata" which is how the plone.app.tiles edit form controls when the overlay it appears in should close.

And finally we need a bit of CSS to style the tiles and edit links:

.tile-editable {
    position: relative;
    outline: 2px dashed #e8e8e8;
    min-height: 1.5em;
}

.tile-editable:hover {
    outline: 2px dashed #b8b8b8;
}

.tile-edit-link {
    display: none !important;
    position: absolute;
    right: 1px;
    bottom: 1px;
    z-index: 500;
}

.tile-editable:hover .tile-edit-link {
    display: block !important;
}

Adding a tile

Okay, now let's add one of these to a template. Pick your favorite template and add:

<div tal:attributes="data-tile string:${context/absolute_url}/@@groundwire.tiles.richtext/hello-world" />

Here's what it looks like in my instance (I added it to the document_view template):

And here's the editing interface that shows up when I click on the pencil:

(If you want to see how this code all comes together, look at the code in https://groundwire.devguard.com/svn/public/groundwire.tiles/branches/davisagli-blocks)

In conclusion

We are very happy with the way the tile approach turned out for the Net Impact site. Once we had mastered the basic technique for landing pages, we soon realized that tiles provided a useful way to add user-editable content areas anywhere in the site. Site needs a doormat in the footer? Use a tile with the Plone site as its context so it appears the same throughout the site and the client can edit the links. Client wants a block of text they can edit on the login form to promote registering for the site? No problem, just add a tile. Client is repeatedly asking for minor edits to the text introducing a custom form? No problem, we turned it into a tile and told them how to edit it. Since the presentation of tiles in the UI is simple and consistent, the barrier to entry for the client to learn how to edit a new tile was very low.

The approach as described here isn't perfect. One thing that needs some care is cache invalidation. In our case, we wrote an ObjectModified event handler for tiles that updates the modified time of the page on which the tile appears. Another limitation is that text in tiles won't be included in the fulltext index unless you go to extra lengths. Whether that's a feature or a bug depends on your use case.

Overall though, we love the technique and have also started using tiles in other sites. I know that Six Feet Up has also successfully used tiles with at least one client. If you want to expand your Plone layout repertoire without using experimental technology like Deco or removing control over content from your clients, I encourage you to give it a try!

When the pull request involves a single commit, it's pretty straightforward: I merge the pull request to master through github's UI, or via the command line with git merge. Then I check out the maintenance branch and use "git cherry-pick" to apply the changeset from the commit relative to the older branch.

But this quickly gets annoying if the branch I'm trying to merge involved multiple commits, and I have to cherry-pick each one in turn. So here's a different approach I used yesterday to backport a changeset from zedr (Rigel di Scala) to add an Italian translation to plone.dexterity.

His pull request was against the master branch, so I first merged that using the github UI.

Then I needed to apply the same change to the 1.x branch of plone.dexterity.

In my local copy of the plone.dexterity repository, I added zedr's fork as a remote, and fetched it.

$ git remote add zedr https://github.com/zedr/plone.dexterity.git
$ git fetch zedr
From https://github.com/zedr/plone.dexterity
 * [new branch]      1.x        -> zedr/1.x
 * [new branch]      davisagli-extend-fsschema -> zedr/davisagli-extend-fsschema
 * [new branch]      jbaumann-locking -> zedr/jbaumann-locking
 * [new branch]      jmehring-drafts -> zedr/jmehring-drafts
 * [new branch]      master     -> zedr/master
 * [new branch]      toutpt-unicode -> zedr/toutpt-unicode

Next I created a new branch called "zedr-merge" specifically for carrying out the merge, based on zedr's forked master branch which contained the change. I needed a temporary branch for this in order to carry out the rebase in the next step.

$ git co -b zedr-merge zedr/master
Branch zedr-merge set up to track remote branch master from zedr.
Switched to a new branch 'zedr-merge'

Now for the fun part. I used rebase to modify the zedr-merge branch's history so that it contains the commits from the 1.x branch, followed by only the existing commits from the zedr-merge branch that I wanted.

$ git rebase -i --onto 1.x a36f40743d67da8e6d5c7b0aee81e786a2de9f5e

Let's break this down. The rebase command will first store all commits on zedr-merge from a36f40743d67da8e6d5c7b0aee81e786a2de9f5e to HEAD in a temporary location (I found this hash using git log to identify the last commit prior to the changes I was trying to backport). Next it resets the history and state of the zedr-merge branch to be equivalent to that of the 1.x branch (because of the "--onto 1.x"). Finally it reapplies the changes that were stored in the temporary location. The end result is that we have exactly the history we want—that is, the 1.x history plus the relevant portion of the zedr/master history—but it is on the zedr-merge branch rather than on 1.x where we want it to end up. We'll deal with that in a moment.

Notice one more thing about the rebase command. I used the -i flag, which means interactive rebase. This means that I'll be prompted in an editor with a list of commits, and can choose to "pick" (include), remove, or "squash" (merge into the prior commit) each commit. In my case, since zedr had a number of commits making small changes to his translation file that were really all part of the same change at the macro level (adding the Italian translation file), I squashed them all together so that I ended up with a single commit at the HEAD of the zedr-merge branch which accomplished the same changes as all of the commits zedr had made on his master branch.

pick 8562549 Added Italian translation
squash d035596 Correct timestamps
squash 9fa9904 Removed template header
squash 4a97700 cancel does not really mean that; fixed (thanks gborelli)
squash aef40f1 messags now coherent with the standard Italian translations found elsewhere
squash 621e185 Updated changelog

# Rebase a36f407..621e185 onto 9408d8d
#
# Commands:
#  p, pick = use commit
#  r, reword = use commit, but edit the commit message
#  e, edit = use commit, but stop for amending
#  s, squash = use commit, but meld into previous commit
#  f, fixup = like "squash", but discard this commit's log message
#  x, exec = run command (the rest of the line) using shell
#
# If you remove a line here THAT COMMIT WILL BE LOST.
# However, if you remove everything, the rebase will be aborted.

At this point, since I had a single commit that I cared about on zedr-merge, it was a simple matter to use git cherry-pick to apply it to the 1.x branch instead.

$ git co 1.x
$ git cherry-pick 6dec9429d7994f02e332e22f8009a687ee3944c0

And now git log shows the change all together nicely in one commit:

$ git log 1.x
commit 6dec9429d7994f02e332e22f8009a687ee3944c0
Author: zedr <zedr@zedr.com>
Date:   Mon Nov 21 12:29:32 2011 +0100

 Added Italian translation
 
 Correct timestamps
 
 Removed template header
 
 cancel does not really mean that; fixed (thanks gborelli)
 
 messags now coherent with the standard Italian translations found elsewhere
 
 Updated changelog

This is really the story of my first realization of the power of git rebase. But one word of warning: you should not rebase commits if they have been shared (i.e. by pushing to github) and others may have based further work on them. It can lead to duplicate commits in the history if the derivative branch is later merged as well. In my case this is not a concern, though, since the maintenance branch will never be merged back to master.

I really don't know if this is the best method for my use case, but it at least had the end effect I was going for in this case. I'm still not quite sure what the simplest approach would be if I wanted to end up with all the commits from zedr/master separately in the history of the 1.x branch, instead of squashing them. I'd be interested to hear if other people are using different approaches.

The concept

Matthew first approached me about a month ago with the intent to pull off something epic for April Fool's day this year. His goal, he explained, was to "make something that nutcases would find useful but everyone else knows is stupid." We discussed various ideas such as supporting ymacs as a richtext editor in Plone before I remembered I had seen a way to run Python in the browser. We quickly ruled out running all of Zope2 in the browser as too big a project, but Matthew suggested doing just the ZODB, and I realized that making it be backed by HTML localstorage could make for a fun, reasonably scoped, buzzword-compliant demo. The idea was born.

The Emscripten Python interpreter

The hardest part of the problem—getting a Python interpreter implemented in Javascript—was already basically solved by the Emscripten project. Their Python interpreter was generated by compiling CPython to LLVM bytecode using clang, then using their tools to translate that into Javascript. The result is a 2.8MB closure-compiled "python.js" which includes the logic of CPython as well as implementations of basic C library calls like sprintf and malloc in terms of operations on a heap which consists of a Javascript array. We didn't have time to get the whole Emscripten toolchain set up and working so that we could build a non-packed python.js, but we did need to understand the basics of this Python interpreter, so we used the Google Closure Compiler to unpack the whitespace of python.js so it was at least semi-readable.

Unfortunately this Python interpreter had a major limitation—it had no implementation of importing modules, so you were limited to things like "sys" which are included statically in CPython. Obviously this wouldn't work for getting the ZODB working.

The import system

I wanted to allow dynamically importing as many things as possible, rather than simply bundling the ZODB code with the interpreter in some fashion. So it seemed like a good approach would be to write a small WSGI import server, and then make the interpreter fetch imports via AJAX in some way. But how, exactly?

I knew that importing in Python calls the __import__ builtin, so I could monkeypatch __builtins__.__import__ to make it somehow fetch the module being imported by name, then manually construct a module using imp.new_module() and exec the fetched code in the new module's namespace. However, this would be Python code running within the sandbox of the Javascript-based interpreter, without the ability to make Javascript calls like fetching via AJAX. So how could we do the actual "fetch the code" step?

We could have used the CPython API (as translated into Javascript) to, from Javascript, create a new "builtin" module with a function for loading a module's code via AJAX. But, neither of us had worked with the Python C API much, let alone with its closure-compiled Javascript variant, and this seemed like too big of a task. So we hit on a simpler hack: we wrote a Javascript function to do the fetching and then "hijacked" the raw_input builtin by replacing the interpreter's reference to it (we picked raw_input because the Emscripten Python interpreter didn't implement it anyway).

The result is a glorious mixture of CPython API (we had to use a bit after all to unpack the argument with the module name and to pack the string with the returned source code) and JQuery:

function raw_input(self, args) { 
    // stack management
    var b = a;
    a += 4;
    for(var d = b;d < a;d++) {
        i[d] = j[d] = 0
    }
    i[b] = 0;
    // unpack argument
    Module._PyArg_UnpackTuple(args, $ba, 0, 1, u([b, 0, 0, 0], 0, o));
    var name = ma(Module._PyString_AsString(Module._PyObject_Str(i[b]))); 

    // fetch via *synchronous* XMLHTTPRequest
    output('Importing ' + name + '...', 'status');
    var source = '';
    jQuery.ajax({
        url: 'lib/' + name,
        error: function(xhr, status, code) {},
        success: function(result) {
            source = result;
        },
        async: false,
        dataType: 'text',
        cache: true
    });

    // return the source as a pointer into the Python heap
    var h = Module.Pointer_make(Module.intArrayFromString(source))
    a = b;
    return Module._PyString_FromString(h);
}
// hijack the raw_input builtin
n[RMb] = Module._builtin_raw_input = raw_input;
 

The __import__ hook could then be implemented in terms of the new raw_input builtin:

import sys, imp
_known_bad = set()
def __import__(name, globals={}, locals={}, fromlist=[], level=-1):
    if name in _known_bad:
        raise ImportError('Could not fetch module %s from server.' % name)
    if name in sys.modules:
        return sys.modules[name]

    # call our hook (we hijack raw_input below)
    source = raw_input(name)
    if not source:
        _known_bad.add(name)
        raise ImportError('Could not fetch module %s from server.' % name)

    m = imp.new_module(name)
    m.__file__ = name
    sys.modules[name] = m
    if '.' in name:
        parent, basename = name.rsplit('.', 1)
        if parent in sys.modules:
            setattr(sys.modules[parent], basename, m)
    exec source in m.__dict__
    return m
__builtins__.__import__ = __import__

This Python is included inline in the HTML, and found and executed during initialization of the interpreter. It is a bit buggy in its handling of packages, but worked well enough to let us move on to the more interesting aspects of the project.

Making the ZODB work

So we could import things. "import this" worked great. The ZODB? Not so much. You see, we soon found out that Emscripten's Python interpreter is really quite minimalistic in its builtin modules. "os" is not included, as sandboxed Javascript can't access the local filesystem, so anything like "logging" which depends on it was a problem. Things like "threading", "re", and "time" were similarly missing. Even more problematic was the omission of the following modules which are used in pickling (sort of the core function of the ZODB): cPickle, marshal, and struct.

So we started hacking up our copies of the ZODB and transaction packages. We removed all the logging. We took out the threading locks, with the justification that Javascript is single-threaded anyway. time.time() got replaced with a simple incrementing counter. Et cetera. As for cPickle and its dependencies, we borrowed the pure Python implementations from PyPy. We also needed Tres Seaver's branch of ZODB to provide a pure Python implementation of the 'persistent' module. It took a couple evenings, but without too much effort we were eventually able to instantiate a DemoStorage, instantiate a DB, connect to it, and commit transactions on the root object. Major win!

The HTML localstorage backend

But it still wasn't a great demo. We wanted it to be possible to commit a transaction, then come back after leaving the page and be able to access the data that had been committed. And we wanted the persistence to happen in browser localstorage on the client side, rather than by passing values to the server. So we needed to find a way to modify or replace DemoStorage to pass its values to Javascript to be placed in localstorage, and to retrieve them again when the page is loaded.

After the CPython API hackery needed to get the imports working, I was a bit scared about doing a lot of passing values from Python to Javascript and back. So at this point I thought, "Wait. We have the entire Python interpreter runtime state in these Javascript arrays; why don't I just save and restore the whole interpreter?" Ultimately this led me down a rabbit hole to nowhere. I never quite figured out the correct bootstrapping process to get all the necessary Javascript variables re-initialized on subsequent loads, but with the Python heap, stack, etc replaced with the old state. And I was bumping up against the 5MB limit for what can be placed in localstorage.

Fortunately Matthew came along at this point with a different approach. He wrote a very simple ZODB storage class, the HTML5Storage (code), which stores pickles of modified objects and writes them to a (Python) global dict, keyed by object id, when a transaction is committed. And instead of messing around with the CPython API to interface Python with Javascript, he simply made a commit print out the JSON representation of that global dict, with a special identifier that the Javascript implementation of print() was modified to watch for and handle specially by parsing the JSON and stuffing it in localstorage. When the page is loaded, the stored values are passed back to Python by converting the localstorage contents to JSON, executing it as Python, and placing the values back in the Python global dict. (There is a bit of extra hackery to encode backslashes in the Python repr of the pickles, handled by the very Britishly named dodgy_encode function.)

At this point, it should have worked. But there was one more hurdle.

Debugging the pickles

When we tried to reload the root object from the HTML5Storage, we were getting unexpected errors. I compared the pickles that had been generated in the Emscripten interpreter with those generated for a similar object on a real Python interpreter, and noticed that they were not the same. I used the pickletools.dis() function from the stdlib to examine the pickle bytecode, and figured out that the size of some strings in the pickle was getting recorded incorrectly, so the pickles were not being executed correctly during unpickling. Specifically, the size of some strings was getting recorded as \x02 regardless of the actual length of the string.

I tracked the bug down to the repr() of the pickle that Matthew was doing in his dodgy_encode function. A variety of different bytes were all getting repr'd as \x02. And then I tracked this into the PyString_Repr implementation of the Javascriptified Python interpreter, to where it calls sprintf with a format of "\\x%02x". It turns out that Emscripten's implementation of sprintf was incomplete and supported neither the "02" used immediately after % to give a zero-padding width, nor the "x" used to specify printing a hex value.  It was interpreting the "02" literally, and the x was getting truncated off. Once I figured out where this issue was arising, it was a relatively simple matter to adjust the sprintf implementation to handle this format correctly.

And there was pickling, and there was unpickling: the first day. And Matthew and David looked on the hackery and saw that it was good.

The launch

Fortunately we still had a day left to put a bit of polish on the thing before April 1. Ryan Foster was kind enough to whip up a nice web-2.0-product-style design on short notice. (Kudos to Ryan; I basically said "we want something like evernote.com" and he magically figured out exactly the sort of color scheme and logo I had in mind.) We added a bit of varnish in front and added some social media bling to the footer. In a flash of inspiration, I dubbed the site a project of "POSKey Enterprises," a reference to the POSKeyError one gets when trying to load an object from the ZODB that does not exist. We revamped the input/output to be much nicer and closer to a real Python interpreter.

zodb.ws launched to much fanfare on the morning of April 1. Like, I mean, we got literally dozens of pageviews. (Okay, actually a few hundred. And we realize the thing is a bit esoteric.) :) Some people assumed the thing was just a light frontend to a server-side interpreter, so people were more impressed once we explained that everything was actually executing client-side. Thanks to everyone else who retweeted the link for helping spread the word a bit beyond the tiny Zope circles!

Ultimately, I think the project satisfied our goals; after all, hacking is about the journey, not the destination.

The result is easy to install and looks like it will be fairly useful for seeing all the objects present in a ZODB (not just the ones that the ZMI or some other app-level tool chooses to show). As an added bonus, it knows how to browse "broken" objects, so you don't have to have your application code in Eye's PYTHONPATH.

(Blue items are persistent objects; black ones are included in the ZODB only by virtue of being referenced by persistent objects, and do not get their own pickle.)

Eye can also be used to take a peek at any old set of Python objects that are not in a ZODB.

See the PyPI page for installation and usage instructions, or clone the project on github and send me pull requests. :)

At Groundwire it is very common for us to run a number of Plone sites within the same Zope instance. This introduces some unique requirements for add-on products to follow when they are registering Zope components:

1. If the add-on registers some new component, it should do it in such a way that it is only available within sites that have the add-on installed.
2. If the add-on overrides some default component from Zope or Plone, it should do it in such a way that it can be further overridden for one particular site.

There are several common approaches that can help with these requirements, but each has downsides. Overriding components using overrides.zcml is global (i.e., affects all Plone sites in an instance) and prevents further customization. Registering components for a browser layer only works for components that adapt the request (such as browser views). Registering persistent local utilities or adapters in a site's local component registry keeps things isolated, but can be a headache when it's time to uninstall the add-on or remove the implementation of a component.

There is a lesser-known fourth option: using z3c.baseregistry to create a registry specific to one add-on.

Component registries in Zope 2

In Zope 2 we typically deal with 2 types of component registries (also called site managers historically):

1. The global registry, which is populated with components at startup by processing ZCML.
2. A local registry associated with each Plone site (implemented in five.localsitemanager). These store components persistently in the ZODB, and can be populated via the zope.component.interfaces.IComponentRegistration API in Python, or via GenericSetup (componentregistry.xml)

When Zope traverses over a Plone site, its local registry is set as the active registry (via zope.site.hooks.setSite—or zope.app.component.hooks.setSite in older Zopes—which sets a thread local). After that, this registry is the one that can be obtained by zope.component.getSiteManager, and the one that will be implicitly used by the functions that do component lookups. If a component is looked up but not found in the local registry, it will fall back to checking in that registry's base registries. By default in Plone, there is just one base registry, which is the global registry.

Introducing z3c.baseregistry

However, there's no requirement that the global registry be the only base registry. z3c.baseregistry makes it possible to define additional, named registries which can be installed as additional base registries for a particular site. Then when a component lookup occurs, it will be looked for first in the local registry, then in the custom base registry, and then in the global registry.

The cool thing is that while the installation of a z3c.baseregistry is persistent, the components one contains are not. Instead, the components are populated at Zope startup via ZCML, very much like the global registry. The registerIn grouping directive lets us specify which registry components should be registered in:

<registerIn registry=".packageComponents">
  <!-- component directives here -->
</registerIn>

This means that when you uninstall an add-on that has its own base registry, you just need to remove the registry from the site manager's bases, rather than figuring out how to unregister each individual component as would be necessary for persistent components in the local registry. It also means that you can safely remove a component's class when you remove its registration without worrying about breaking legacy persistent registrations of that component.

Step by step

I've used this approach in a few projects lately. Here's what it looks like:

1. Add z3c.baseregistry to the add-on's install_requires in setup.py, and re-run buildout to make sure it is installed.

2. Create a new registry instance.

   In __init__.py (or could be elsewhere):

   from zope.component import getGlobalSiteManager
   from z3c.baseregistry.baseregistry import BaseComponents
   packageComponents = BaseComponents(getGlobalSiteManager(), 'foo.bar')

   Here, we made sure that the new registry has the global registry as its base, and is named after our add-on package (foo.bar).

3. Register a local utility for looking up the new registry by name (this is used by z3c.baseregistry internally).

   In configure.zcml:

   <!-- registry for package-specific components -->
   <utility
    component=".packageComponents"
    provides="zope.component.interfaces.IComponents"
    name="foo.bar"
    />

4. Install the new registry in the bases for a particular site. z3c.baseregistry includes a form for doing this through the web, but it doesn't seem to work in Zope 2. Oh well, we can do it with a GenericSetup import handler instead.

   In setuphandlers.py:

   from zope.component import getSiteManager
   from zope.component.interfaces import IComponents
   
   def install_base_registry(site):
       sm = getSiteManager(context=site)
       reg = sm.getUtility(IComponents, name=u'foo.bar')
       sm.__bases__ = tuple([reg] + [r for r in sm.__bases__ if r is not reg])

   You would then call this from the add-on's custom "import various" GenericSetup handler.

5. Now components can be registered for the new add-on specific registry, using the registerIn grouping directive.

   In configure.zcml:

   <!-- make sure we can use registerIn -->
   <include package="z3c.baseregistry" file="meta.zcml"/>
   
   <registerIn registry=".packageComponents">
   
     <browser:page
       for="*"
       name="foobar"
       template="foobar.pt"
       permission="zope.Public"
       />
   
   </registerIn>

   These components will be found within sites that have the product installed, but not within sites that don't!

Caveats

• It should be obvious, but this only localizes the effects of ZCML directives whose effect is to register something in the component registry (e.g. utility, adapter, subscriber, browser:page). Directives that mutate other things, such as <class> which directly modifies a class, will still have a global effect.
• Don't forget to make sure that the base registry gets removed from the local registry's bases when the add-on is uninstalled. Otherwise removing the product will break the site when it tries to unpickle the base registry.

(Thanks to Darci for the screen capture.)

You can still see the results yourself, although Plone is now second after a blog post about Google's instant search.

See the link above for info on major features in the release, but there are a lot more little things included too. Without further ado, here are ten of my favorite Plone 4 improvements that nobody's talking about...

General / site admin features

1. New editor features. Check the control panel for TinyMCE (our new WYSIWYG editor) to find some useful buttons that aren't enabled by default:

• Paste as Plain Text (preserves newlines)
• Paste from Word (strips junk markup)
• Insert/edit Media

(among others)

2. Better date/time handling. The display of event times in listings is now smarter. If the event begins and ends on different days, the time is not shown. If the start and end times are the same, it is not shown twice.

In addition, dates and times are now stored with a timezone, with proper consideration of daylight savings time.

3. Inline error tracebacks. If you're logged in as a Manager and encounter an error, the traceback is shown immediately, rather than requiring you to click through to the error_log in the ZMI.

4. Permission auditing. Thanks to a new feature in Zope 2.12, it's now easy to double-check what permissions a particular user will have in a particular context, via this button on the Security tab in the ZMI:

For integrators

5. Better control over portlet columns. In Plone 3, it was possible to make a template prevent rendering of the portlet columns by filling METAL slots. In Plone 4, this can be done by setting the disable_plone.leftcolumn and disable_plone.rightcolumn flags on the request before calling main_template. For example:

<tal:block tal:define="foo python:request.set('disable_plone.leftcolumn', 1)"/>
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en"
 xmlns:tal="http://xml.zope.org/namespaces/tal"
 xmlns:metal="http://xml.zope.org/namespaces/metal"
 xmlns:i18n="http://xml.zope.org/namespaces/i18n"
 lang="en"
 metal:use-macro="context/main_template/macros/master"
 i18n:domain="plone">

(snip)
</html>

(This might also be done in Python in the __init__ method of a browser view.) Handily, it's actually a 3-way flag: set it to True to force hiding the colum, False to force displaying it even if there are no portlets assigned, and None to use the default logic based on whether there are portlets assigned.

6. "All content" listing view and content-core macro. There is a new folder listing display option called "All content", which renders the full content of each item. (See it in action on the homepage of this website.)

This listing takes advantage of the new content-core slot which is defined in main_template following the title and description and between the various above/below viewlet managers, and filled by each content type's default view. This gives each content type control over how it renders in the "all content" listing. (The "all content" listing also takes advantage of some new slots/macros in the folder_listing template to reuse its logic for batching and fetching item properties; this may be a useful model for other custom folder listings.)

It also makes it simpler to write a view template for a content type, since one no longer needs to include the title, description, and common viewlet managers as boilerplate. The simplest view template is now (this is a copy of Plone 4's document_view):

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en"
 xmlns:tal="http://xml.zope.org/namespaces/tal"
 xmlns:metal="http://xml.zope.org/namespaces/metal"
 xmlns:i18n="http://xml.zope.org/namespaces/i18n"
 lang="en" metal:use-macro="context/main_template/macros/master"
 i18n:domain="plone">
 <body>
  <metal:content-core fill-slot="content-core">
   <metal:content-core define-macro="content-core">
    <metal:field use-macro="python:context.widget('text', mode='view')">
     Body text
    </metal:field>
   </metal:content-core>
  </metal:content-core>
 </body>
</html>

For backwards compatibility, templates may still continue to fill the main slot instead of content-core, and provide their own title, description, and viewlet managers.

7. Flexible image scaling. There is now an Image Handling control panel which allows configuring the standard image scale sizes used by Plone.

In addition, since image scales are now generated on demand, it's quite easy for a template to request an image scale of a non-standard size, using this idiom:

<img tal:define="scale context/@@images"
     tal:replace="structure python: scale.scale('image', width=42, height=42).tag()" />

Including a standard image scale is simpler too:

<img tal:replace="structure context/@@images/image/mini" />

See plone.app.imaging for additional options.

For developers

8. Automatic registering of Zope 2 permissions. In Zope 2.10, registering a new permission took two steps (using the <permission/> ZCML directive to register a Zope 3 permission utility, and registering it as a Zope 2 permission, usually by calling Products.CMFCore.permissions.setDefaultRoles during product initialization).

In Zope 2.12, only the first step is required.  The equivalent of setDefaultRoles is automatically called after processing the permission directive, and sets the default roles to ('Manager',).

If you want different default roles, you must still call setDefaultRoles yourself, prior to loading of ZCML (e.g., at import time). This will improve in Plone 4.1 / Zope 2.13, where the permission directive gets an optional role subdirective to specify the desired default roles in ZCML.

9. Fixed ordering of CMF/Archetypes interaction during object creation. In Plone 3, when you created a new content item, CMF called the factory method of an Archetypes content type, then called its ObjectCreated event, then updated the item's portal_type attribute. This had the unfortunate effect that if you were trying to create a schema extender that used the portal_type to determine whether the extender should be applied (perhaps based on some configurable property), and your site used content types that shared the class of other content types, you were out of luck.

This is now fixed so that ObjectCreated is not fired until the portal_type is set. I think we also managed to avoid calling some event twice during object creation that was causing duplicate catalog indexing, but I could be misremembering that.

10. Folder ordering adapters. The order of items in a folder is now looked up via a named adapter of the folder to plone.folder.interfaces.IOrdering—and it is looked up dynamically when ordering catalog results by getObjPositionInParent, rather than consulting an index. This means that it should now be simpler to customize how a particular folder is ordered (e.g. to automatically sort alphabetically or by date), by writing a new adapter that provides IOrdering and setting a folder to use it. To set the name of the IOrdering adapter that is used for a particular folder, use its setOrdering method.

Out of the box, most folders use the DefaultOrdering adapter, which allows user-configurable order. This should be appropriate in many cases. Folders that were migrated from Plone 3's Large Plone Folders use the "unordered" ordering (but this shouldn't be needed for new folders that will store many items, as the default ordering is reasonably performant). plone.folder also has a "partial" ordering which allows specifying the order for some but not all subitems, but this is not directly used or supported in the Plone UI at this time.

On to 4.1 and beyond!

And Windmill was quite a bit easier to get working. I just added the following to my package's setup.py to define a new installation "extra":

extras_require = {
    'test': ['niteoweb.windmill',],
},

And then modified my buildout's test runner to include that extra:

[test]
recipe = zc.recipe.testrunner
eggs =
    ${instance:eggs}
    plone.app.skineditor [test]
defaults = ['--exit-with-status', '--auto-color', '--auto-progress']

I added a new test module called test_integration.py and made it use the base test case from niteoweb.windmill:

import unittest
from niteoweb.windmill import WindmillTestCase
from Products.PloneTestCase.setup import setupPloneSite
from Products.PloneTestCase.layer import onsetup
from Products.Five.zcml import load_config
from Testing import ZopeTestCase as ztc

@onsetup
def load_zcml():
    import plone.app.skineditor
    load_config('configure.zcml', plone.app.skineditor)
    ztc.installPackage('plone.app.skineditor')

load_zcml()
setupPloneSite(products=['plone.app.skineditor'])

class SkinEditorIntegrationTestCase(WindmillTestCase):

    def afterSetUp(self):
        """Setup for each test
        """
        ztc.utils.setupCoreSessions(self.app)
        self.setRoles(['Manager'])
        self.login_user()

    def test_customize_logo(self):
        import pdb; pdb.set_trace()

def test_suite():
    return unittest.defaultTestLoader.loadTestsFromName(__name__)

(Most of this is standard test setup boilerplate. I could probably be a bit more sophisticated about the test setup and use layers or something, but this is the tried and true test setup I've been using since Martin published it in his book. The load_zcml method and setupPloneSite method are deferred and run when the Plone test layer is set up by the test runner.)

So far I just added one test that enters pdb. At this point, I can run the test with bin/test -s plone.app.skineditor, and Windmill will start up the ZServer with a dummy Plone site, fire up a browser and a controller window, and then pause at the pdb. (Unlike Selenium, I don't have to install a browser plugin or have another process running first. Nice!) Then I can play around with the controller and start recording tests.  The afterSetUp method runs before each test, and calls niteoweb.windmill's "login_user" helper, which does an initial login to the site as a Manager user.  It also tells the test infrastructure to support sessions, which are needed for my app.

Actually recording a test is mostly a point-and-click affair—hit the "record" button and go at it—but with a few caveats that I'll note below. It took a little bit for me to get used to the ways of selecting parts of the page and making assertions, but at least Windmill provides some flexibility. You can select by id, via XPath expressions, via JQuery selectors, or several other methods.  And assertions range from asserting that certain text is on the page to asserting that an arbitrary Javascript expression evaluates to true.

Here's the full test case I ended up with (dumped from the controller using the "save" button):

def test_customize_logo(self):
    client = self.wm
    client.click(id=u'user-name')
    # load customizer
    client.click(link=u'Site Setup')
    client.waits.forPageLoad(timeout=u'20000')
    client.click(link=u'Theme Editor')
    # go into advanced mode and customize based on the logo in the
    # non-active "plone_images" layer (Windmill doesn't do file uploads)
    client.click(link=u'Advanced')
    client.click(id=u'plone-app-skineditor-name-field')
    client.type(text=u'logo', id=u'plone-app-skineditor-name-field')
    client.click(id=u'plone-app-skineditor-filter-button')
    client.waits.forElement(timeout=u'', id=u'plone-app-skineditor-browser')
    client.click(xpath=u"//a[@id='skineditor-logo.png']/dt")
    client.waits.forElement(xpath=u"//dd[@class='plone-app-skineditor-layers']")
    client.click(jquery=u"('a[href*=plone_images/logo.png/manage_main]')[0]")
    client.waits.forElement(jquery=u"('#pb_1 input[value=Customize]')")
    client.click(name=u'submit')
    client.waits.forElement(timeout=u'', id=u'pb_2')
    # now reload and make sure the logo has the height we expect from the
    # customized image
    client.refresh()
    client.asserts.assertJS(js=u"$('#portal-logo').height() == 57")
    # now remove the customization
    client.click(id=u'plone-app-skineditor-name-field')
    client.type(text=u'logo', id=u'plone-app-skineditor-name-field')
    client.click(id=u'plone-app-skineditor-filter-button')
    client.waits.forElement(timeout=u'', id=u'skineditor-logo.png')
    client.click(xpath=u"//a[@id='skineditor-logo.png']/dt")
    client.waits.forElement(xpath=u"//dd[@class='plone-app-skineditor-layers']")
    client.click(link=u'Remove')
    # and confirm we're back to the original height
    client.refresh()
    client.asserts.assertJS(js=u"$('#portal-logo').height() == 56")

And here's what it looks like to run that. (No clicking on my part involved!) The excitement starts about 0:17...

Finally, here are a few caveats I ran into while setting up my first test, to customize the logo using plone.app.skineditor, and some last observations.

• Windmill can't do file uploads. This is a limitation of browser Javascript support / sandboxing, not of Windmill per se.  It would be nice if there were some command that would prime the Windmill HTTP proxy to add a particular file to the next HTTP request that comes through, so that uploads could at least be faked.
• Windmill could be a bit smarter about handling AJAX requests when recording.  I needed to manually add a "waitForElement" step after each click that resulted in an AJAX load, to make sure that the load was complete before subsequent steps run.  It would be nice if there was a "wait for all ajax to complete" command so that I didn't have to identify a particular element to wait for.
• I wish that there was a wrapper that would let me control Windmill using the same API as zope.testbrowser, to make it easier to convert existing tests to full browser tests.
• So far I've only tried running tests in my default browser (Firefox), but it's possible to run in other browsers as well.  The niteoweb.windmill page on PyPI gives an example of a test layer for doing this.
• It remains to be seen how cumbersome this sort of test will be to keep up-to-date as the product evolves.

Overall Windmill provides a decent experience for recording tests and a really nice one for running them. Perhaps it is a tool we can use to do real browser testing for Plone core?

If you've used Windmill and have any insights into how to use it effectively, I'd love to hear your thoughts in the comments.

Thanks to the great work of my fellow sprinters at Plone Symposium East, I'm now ready to give a preview of the tool, which hopefully will see a first beta release real soon now™. Here's the screencast...

I forgot to mention in the screencast that Eric Steele has also started work on integrating this tool with Gloworm, so that it will be possible to find a resource to customize just by pointing and clicking (a la Firebug).

Currently the customizer displays items from CMF skin layers, browser view templates, viewlets, and portlets. Support for other things like browser resources or ZMI pages could probably be added, at least in a read-only fashion. The infrastructure is flexible enough to support new ways of registering resources that haven't been invented yet.

For now, if you're adventurous and want to try out the customizer, get a copy of the Plone 4 coredev buildout, and run it using -c experimental/skineditor.cfg

We are tracking bugs and ideas for improvement in the Plone bug tracker, with the component set to "Skin Editor."

This is only the beginning...David Bain is working on the customizer as part of the Google Summer of Code, and we've got lots of ideas about how to make it even more useful.

*better names hereby solicited :)

As a result of a mostly tongue-in-cheek conversation on IRC with Elizabeth Leddy, Alex Clark, and Matthew Wilkes, I decided to set up the global zc.buildout threat level indicator.

The indicator collects reports of success and failure from actual buildout runs, and displays the current threat level based on the percentage of buildouts that have succeeded in the past 4 hours.

I'm happy to report that the current threat level is LOW.

Creating the indicator was mostly an excuse to try writing a Google App Engine app, but it's already proved its utility.  Last Thursday the threat level reached ELEVATED after a new distribute release started causing buildouts to fail under Python 2.4. Thanks to Tarek Ziadé for the quick response with a new distribute release to fix that.

You can help make the indicator more accurate!  Just add the buildout.threatlevel extension to your buildout:

[buildout]
extensions = buildout.threatlevel

This will display the current threat level as one of the initial steps when you run buildout (so that you have an opportunity to abort if it has reached SEVERE), and will ping my app engine app when the buildout is done to report success or failure. (No data is collected in this ping aside from a boolean indicating success, time, and IP address.)

Code for the GAE app and buildout extension is in the collective.