Skip to content. | Skip to navigation

Personal tools

>>> ''.join(word[:3].lower() for word in 'David Isaac Glick'.split())

‘davisagli’

Navigation

You are here: Home

David Glick – Plone developer

by admin posted Apr 05, 2010 11:48 PM

report from the Berlinale sprint: how we can fix the Plone skin situation

by David Glick posted Feb 23, 2009 03:57 AM

Well! I am back from two wonderful weeks of traveling in Germany, Belgium, and Spain, and it's high time that I write something about what I got done at the Berlinale sprint which was a highlight of the trip and is already almost two weeks past!

I headed into the sprint with the goal of figuring out how to make Plone 4 not have two different ways of registering templates and resources, like we have in Plone 3. By the end of the week I had written only a little bit of code toward this end, but I was able to spend a lot of time talking to Hanno Schlichting, Laurence Rowe, and Andi Zeidler and figuring out how we'd like things to work. I feel quite good about the progress we were able to make toward a clearer vision of "what good looks like" in this area, and have a better idea now about what further code needs to be written to make that vision a reality.

To summarize our overall conclusion... I've heard a number of people say "skin layers should die." But this is a bit shortsighted, and probably reflects more the fact that people are tired of having two different ways to do this stuff, and that Zope 3 views and resources are a newer technology, rather than any clear superiority in functionality of the Zope 3 approaches over the alternative. So instead of ditching portal_skins, we should try to modernize it and make it compatible with the Zope 3 approaches.

But, this is a complicated thing to get right, and I would like to have feedback on whether we are on the right track and considering all the important use cases, so let me go into some detail on our current challenge and some concrete steps we can take to improve the situation.

The status quo

Currently in Plone 3 we have two different ways to register a resource that will be accessible via a URL in Zope. There are Content Management Framework (CMF) skin layers, which let you create a directory with a bunch of files (such as templates, images, Python scripts, CMF Form Controller objects, etc.) and associate it with a particular "theme" which may be enabled in your Plone site. These files are then magically available via Python attribute access on the portal root, thanks to a magic __getattr__ method in CMF's Skinnable class, which the portal root class extends.

There are also browser views and browser resources, concepts borrowed from Zope 3. These are generally registered via declarations in the XML-based Zope Configuration Markup Language (ZCML) which result in these resources getting registered as multi-adapters of a context and a request in the global registry (a.k.a. global site manager) of the Zope Component Architecture (ZCA). These adapters are then looked up as one of the steps in the traversal process of the Zope publisher.

Of course, it's confusing to have two different ways of solving the same problem that are both used in different places. (Just try to explain to a Plone newbie what the difference between portal_skins and portal_view_customizations is and when each should be used. There are answers, but they depend mostly on the technical details of the implementation and the history of Plone's development, and are not much use to someone actually trying to use these systems to solve the practical question of how to override component X.) We'd really like to have one system for registering resources. But the problem is that neither of the existing approaches is an obvious winner. Each one has benefits and tradeoffs.

Comparison of CMF skin layers vs. Zope 3 views and resources

A. CMF skin layers / portal_skins
Pros Cons
  • Very easy both to add a new resource and to override an existing one
  • The concept of overriding a resource through the web by copying it to a 'custom' layer is fairly intuitive
  • Supports TTW customization for many different types of resources (templates, images, CSS, JS, KSS, etc.)
  • Includes a diff feature for comparing customized items to the filesystem version.
  • Includes a Find feature for determining which version of an item is active.
  • Tools exist to aid with exporting TTW customizations back to the filesystem.
  • Supports the concept of multiple themes which may be activated at different times
  • Resources are available globally and can collide with content ids
  • The way resources are traversed is a bit magical and may slow attribute access for anything acquired from the portal
  • A bit cumbersome to register a new layer (requires ZCML and Generic Setup)

B. Zope 3 views & resources / portal_view_customizations
Pros Cons
  • Views and resources can be registered more selectively for a particular interface only, so that two resources of the same name can be used in different contexts.
  • Built on infrastructure that is more "core" to Zope (and thus probably better maintained and optimized)
  • Resources can be namespaced by using resource directories, and thereby separated from content.
  • The portal_view_customizations UI is rather poor and doesn't support some of the nice features of portal_skins.
  • Adding a new view or overriding an existing one requires a new ZCML registration.
  • There's no way to override just one item from a resource directory.
  • Through-the-web overrides are only supported for templates.
  • There's no good way to export through-the-web customizations back to the filesystem.
  • No simple way to handle subsite theming...a number of people have been struggling with this lately.

So what do we do?

So how can we continue our modernization toward Zope 3 approaches, without losing all the good things about CMF skin layers?

Based on the conversations I had at the Berlinale sprint, I think the answer involves three parts...

1. Register skin layer items as Zope 3 views and resources

The big coding task I tackled at the sprint was creating a package called 'plone.resource'. This is intended to be a reimplementation of the DirectoryInformation class from CMF (the class which handles exposing filesystem items as objects within Zope) but with different behavior behind the scenes, such that each item in the directory gets registered as a browser view.

Some more details:

  • It will be possible to register new resources (including template-based browser views) just by creating a file in the right directory. If Zope is running in debug mode, we can detect new and removed files without needing a restart.
  • All the items in a particular resource directory will be registered for a particular, configurable Zope 3 browser layer.
  • For templates, additional parameters to the view registration (such as "for" and "name") can be specified either in .metadata files or in <meta> tags in the <head> of the template's HTML. (Of course, ZCML registrations for browser views not located in a plone.resource-managed directory will continue to work.)
  • It will be possible to configure the resources associated with a directory to show up in the global namespace a la portal_skins (for cases where you really do want a resource accessible anywhere), in a ++resource++ namespace directory a la Zope 3 resources (mostly for backwards compatibility), or in a named path like 'images/' (for cases where you want resources separated from content, but want a prettier URL than ++resource++).
  • Because everything gets registered as a browser view in the end, it will be easier for us to create a single UI which can handle customizing all kinds of resources.
  • It should be possible to override existing resources from another package in a fashion similar to z3c.jbot, by creating a file whose name is the full dotted path to the resource being overridden.
  • For now I am reusing the resource classes from CMF; though, as Hanno pointed out, it may be possible to replace these with something more lightweight at some point. Supporting DTML, CMFFormController, and Python scripts in the long term are not priorities for me, although I hope to make plone.resource pluggable enough that support for them and other types of resources could be provided by add-on products.

A lot of this still needs to be built, but the foundation is already in place.

2. Improve our mechanisms for managing how browser layers get activated

Since more things will depend on Zope 3 browser layers, we will need better support in plone.browserlayer for:

  • Making sure that items on a "theme" layer take precedence over items on an "add-on product" layer. (That is, controlling the order in which layer interfaces are applied to the request).
  • Enabling a theme layer only for certain sections of the site (as well as disabling any other theme layer within that section).

I haven't done much thinking yet about the details of either the UI or the implementation for these enhancements. Suggestions welcome.

3. Revamp the UI for customization

portal_view_customizations is functional, but it needs a new user interface...possibly one that is based in the Plone control panel rather than the ZMI. I envision a listing of views and resources driven by a form that lets you filter by name of resource, type of resource, layer it's registered on, interface that a view is "for", etc. (Ideally we could even support a full text search of template contents.) For each resource shown in the results listing, all of the layers providing that resource need to be made evident, along with an indication of which is currently taking precedence. And it should be possible to see a diff of changes for items that have been customized through the web, as well as to export all such items for use in filesystem resource directories.

Alternatively, or in addition, an approach more like gloworm (which operates by the "poke-that-thing-on-the-page-and-tell-me-more-about-it-style" interface) could be taken. As long as plone.resource (or whatever else registers views) takes advantage of zope.configuration's support for annotating the view registration with info about where the registration originated, and as long as we have some way to expose this when a view's template is rendered, this shouldn't be too hard to achieve.

Aside from the new things that need to be built, the main implications this has for the existing plone.app.customerize implementation are:

  • We need to support other types of resources besides templates, both in terms of finding them and storing persistent copies of them. (I need to investigate further whether it's possible to adapterize the function of getting file contents somehow, so that we don't need to have separate persistent and non-persistent versions of each resource class.)
  • When an item gets customized through the web and plone.app.customerize makes a local copy of the view registration, we need to change the layer to something like ITTWCustomized so that it doesn't mask the global registration for the purposes of introspection.

A few last words

Okay, I think that's probably enough to keep me busy for a little while. :) Please don't hesitate to ask questions; this plan incorporates a lot of small improvements and there are no doubt places where I should have been clearer about what something means or why something is a good idea. Constructive criticism and suggestions on how to make this even better are also welcome.

You'll no doubt notice that I haven't said much (if anything) about viewlets and portlets. Don't worry, we know that understanding the distinction between these and learning how to create and register them is one of the common pain points in learning Plone theming today, and we hope to make things easier in Plone 4. But this problem is mostly orthogonal to the question of how resources get registered and customized, and is being dealt with elsewhere, so I'm not going to spend time on it here. For now, suffice it to say that as long as the approach used in Plone 4 for generating a section of a page is based on a Zope 3 browser view, it should be compatible with the improvements discussed here.

2 comments

A tale of a resurrected bug

by David Glick posted Jan 25, 2009 05:36 PM

This is the story of a Friday night spent tracking down a particularly pesky bug in my Plone site. In gory detail.

Crap, it's not working...

I had just finished moving one of our larger Plone deployments to a new host. Everything had gone, with one or two minor exceptions, exceedingly smoothly, and I was looking forward to setting aside the laptop for the rest of the evening. (Yes, that does happen occasionally ;) ...But then I got an IM...

[our client]: 22:06:50 i am getting:

Bad gateway The proxy server received an invalid response from an upstream server

Well, shit. That's weird... Normally I might have expected this 502 error from squid when Zope is restarting, but in this case Zope is directly behind apache, and is running fine. And the error's only occurring on one particular page...what the heck? Also... why is this a problem now when it definitely wasn't happening on the old server? My entire software stack is installed from a buildout with pinned versions, and I copied the apache configuration, so what variables could be left?

I could probably have already known at this point that the evening was shot. But what to do?

Time for some inspection...

Okay, well obviously Apache doesn't like something it's getting from Zope. Maybe a Zope bug of some sort? Let's check the apache error log and see what it tells us:

[Fri Jan 23 22:12:05 2009] [warn] proxy: bad HTTP/1.1 header returned by /@@sub_trial (POST)

Gee, thanks for all the detail, apache. Well, at least this confirms my suspicions that some header from Zope is wrong. But what could it be? Time to hit Zope directly and see what we can see in the headers. Open up the port in the firewall, enter the Virtual Host Monster URL to the problem page, view the headers in Firebug...what have we got?

Date: Mon, 26 Jan 2009 08:35:33 GMT
Server: Zope/(Zope 2.10.5-final, python 2.4.4, linux2) ZServer/1.1 Plone/3.0.3
Content-Type: text/html; charset=utf-8
Location: https://www.xxxxx.org/thankyou_trial
Set-Cookie: statusmessages="A+RXZWxjb21lISBZb3UgYXJlIG5vdyBsb2dnZWQgaW4uaW5mbwTEWW91ciBUcmlhbCBTdWJzY3JpcHRpb24gSXMgTm93IEFjdGl2ZS5pbmZv"; Path=/
Via: 1.1 xxxxx.org
Vary: Accept-Encoding,User-Agent
Content-Encoding: gzip
Content-Length: 20
Keep-Alive: timeout=15, max=100
Connection: Keep-Alive

Well, the page loads fine when I hit the site directly in Zope, and those headers look pretty innocuous. At this point my only guess was that the gzipping was screwing things up somehow, and thus followed a half-hour-long digression of confirming the gzip settings in CacheFu, confirming the mod_deflate settings in apache, and finally disabling gzipping entirely in Firefox. All of which served only to tell me in the end that the problem had nothing to do with gzipping.

Pull out the big guns: Google

At this point I did what any self-respecting software professional would do: I typed the error message into Google. This is where things really start to get interesting...

A query of zope "bad HTTP/1.1 header" netted me a few irrelevant-looking results. But aha, also an old e-mail thread that describes a similar error, and that has a link to a Plone trac ticket that...well, "Plone Hotfix 20071106 breaks long status messages" doesn't exactly sound relevant, but look, there's a comment that mentions the bug results in a 502 gateway error when running behind apache -- which matches my experience -- and it even says the bug is still present in Plone 3.0.3, the version I'm using on this particular site. And the description of the issue -- a problem with the statusmessage cookie header -- sounds reasonably close to the diagnosis I was guessing at. Hey, maybe we're on to something!

But something is still not right. The comments on the ticket also say that a newer version of the hotfix that introduced the bug, PloneHotfix20071106, was released to fix the issue all the way back in November 2007. And my buildout already includes this updated version -- it's right there as:

http://plone.org/products/plone-hotfix/releases/20071106-2/PloneHotfix20071106.tar.gz

(Note the "-2"). Hmm, now that I think about it, maybe I have a vague recollection of encountering this same issue sometime in the distant past, and updating that hotfix in the buildout. But, obviously the fix is not working now. So what's going on?

The epiphany

And then it hit me.

David Glick: 23:14:50 holy crap, I think the real problem here is related to the recent plone.org upgrade David Glick: 23:14:56 no shit :)

Let me explain. During the recent plone.org upgrade, all the product tarball files stored in PloneSoftwareCenter were migrated from being stored in the ZODB to being stored in a big directory on the filesystem using FileSystemStorage and collective.psc.externalstorage. (This is part of the efforts to make PSC function as a pypi-style repository, a cool ability which I hope Alex Clark and Tarek Ziad?? will publicize before too long.) Now, the thing I noticed that made me think "aha!" is that the filename listed for the hotfix tarball in my buildout has no version suffix. And it didn't take me much longer to check the older release of that hotfix and see that it has the exact same tarball filename. And then, knowing what I did about the plone.org upgrade, it didn't take much imagination to think that the two releases must now be accessing the same file on the filesystem, and the "-2" release was getting the wrong data. And this was quickly confirmed by inspection of the actual Hotfix code that had been fetched into my buildout.

To express my feelings at this point, echoing the immortal words of our Zope Pope:

"WAAAA!"

Case closed? You betcha. I stopped using the broken tarball and checked out the correct version of the hotfix from the svn tag instead. After restarting Zope, no more 502 error when I hit the form. And I followed up soon thereafter by replacing the hotfix tarball with the most recent release so that others shouldn't run into this same issue.

Note, however, that there's no guarantee at this point that other packages aren't experiencing a similar issue, so keep your wits about you when installing old-style product tarballs from plone.org. However, I have filed a report and trust that Alex or Tarek will be able to work on it before too long.

Of course, it goes without saying that I was rather lucky to have the familiarity with the plone.org upgrade needed to correctly diagnose the cause of this issue. But otherwise, just another day in the life of a Plone hacker...

PloneFormGen gets a little shinier

by David Glick posted Dec 06, 2008 04:05 AM

Over the past couple days Steve McMahon has been with us at the ONE/Northwest Seattle office, and we've been working on a number of improvements to PloneFormGen. With the involvement of several other Seattle-based Plone developers who often join us for open source work on Friday afternoons anyway, we had a small army working on PFG.

Faced with a long list of good ideas, we decided to focus on the relatively easy changes. We'll try to get these through beta to a final release fairly quickly so that we can move on to more interesting challenges like a revamped form editing UI, multi-page forms, and no longer storing fields as Archetypes objects.

Sprint accomplishments:

 

  • Andrew Burkhalter finished converting the installation over to GenericSetup (work begun by Jesse Snyder at the sprint in DC)
  • I merged a branch I've been working on which adds support for rendering PFG forms from another template, viewlet, or portlet.
  • Steve McMahon cleaned out some compatibility code for Plone 2.1 (which will no longer be supported except in the 1.2.x maintenance series)
  • I created a proof-of-concept of a recaptcha integration which we plan to ship with PFG. This needs more work though.
  • Steve added an option to the mailer to select which fields will be included in the e-mail, as well as an option to exclude empty fields.
  • Steve removed the restriction on which fields can be used as the mailer recipient.
  • Jon Baldivieso added a link spam validator to ensure that a field doesn't contain links.
  • Steve switched the default view of form fields to the edit view, to save a few clicks when building forms.
  • Steve added CSS ids to the rich label field.
  • I fixed up the test infrastructure to work with roadrunner, and added some basic integration tests.
  • Andrew made progress on his branch which will add a form import/export feature.
  • Paul Bugni added an e-mail validator that support multiple e-mail addresses.
  • Alex Tokar and Fulvio Casali almost finished implementing an option to make a field "server side" only, which is like a hidden field that doesn't actually render for the client (to avoid the chance of hacking hidden fields).
  • Steve added a BCC field override setting.
  • Jesse Snyder fixed a bug where copied-and-pasted action adapters were not automatically activated.
  • Michael Dunlap removed the isUrl validator from the form action override, so that it's possible to use relative URLs.
  • Elliot Cohen joined the fun even though he's new to Plone, and reviewed PFG documentation.
  • I added a 'force SSL' option for form folders which will redirect to an https:// version of the form.

Thanks to everyone who participated, including those who responded to my previous request for voting on features!

2 comments
David Glick

David Glick

I am a problem solver trying to make websites easier to build.

Currently I do this in my spare time as a member of the Plone core team, and during the day as an independent web developer specializing in Plone and custom Python web applications.