Skip to content. | Skip to navigation

Personal tools

>>> ''.join(word[:3].lower() for word in 'David Isaac Glick'.split())



You are here: Home

David Glick – Plone developer

by admin posted Apr 05, 2010 11:48 PM

Image captions and the PortalTransforms cache

by David Glick posted Apr 26, 2009 01:30 PM

Plone 3 contains a cool feature whereby images inserted in kupu can automatically be captioned with the description stored on the image itself. However, if you've tried to use this you may have noticed that after you edit the description, and then try to view a page that includes that image with a caption, the caption sometimes does not update for up to an hour.

The reason has to do with the way in which the captions are added, via the html-to-captioned transform (which should be applied to anything with the text/x-html-safe output mimetype, thanks to a special transform policy that is configured for that mimetype). It turns out that PortalTransforms caches the result of a transform for an hour. It stores this result in a volatile (that means non-persistent) attribute of the value that is being transformed, so in normal cases when you are editing and replacing that value entirely, we don't have to worry about cache invalidation--that is, if I save new body text for a page, the cached transform value will be wiped out and we'll see the new text the next time we view the page. However, in the case of editing the caption of an image, I'm saving a completely different value, on a different object, than the one where the transform result is cached. So even if I reload the page I'll see the old caption.

Workaround 1: Change cache lifetime setting

Option 1: Go to /portal_transforms/manage_cacheForm in the ZMI and lower the cache lifetime. But remember that this will have a negative effect on performance (particularly for pages that are viewed frequently but not served out of a reverse proxy), as the transform will have to be applied more often.

Workaround 2: Manually invalidate the cache when the image is edited

Option 2: When the image caption is edited, invalidate the transform cache on the field referencing the image. I took a first stab at doing this for a project currently underway. For this project I have a custom subclass of ATImage and images are always contained within a (folderish) article, so I modified the caption mutator as follows:

    def setCaption(self, value, **kw):
        self.getField('caption').set(self, value, **kw)

        # The result of transforms is cached for up to one hour.
        # Our caption is inserted into article text via a transform,
        # so if we're located within an article, invalidate the
        # (volatile) cache of the article's text.
        parent = aq_parent(aq_inner(self))
        if IArticle.providedBy(parent):
            value = parent.getField('text').getBaseUnit(parent)
            if hasattr(value, '_v_transform_cache'):
                delattr(value, '_v_transform_cache')

(Note: I first investigated using the Cache class from Products.PortalTransforms.cache so that I could simply do Cache(value).purgeCache() ...however, in the current PortalTransforms release the purgeCache method uses a method which was not imported, so I just recreated what that method does. I checked in a fix to the PortalTransforms bug which should make it into the next release of that package.)

For a more generic implementation of this cache invalidation, you would want to do the cache invalidation in a handler for the IObjectEdited event on ATImages, and use whatever the linkintegrity code does to determine what items contain references to the image which was edited--and are therefore candidates for cache invalidation. (The cache invalidation is only needed if the referencing field has already been rendered and is still in memory in the ZODB cache, so of course it would be good to make sure that the cache invalidation doesn't cause extra objects to be woken up.)

I don't have the need, time, or interest to work on this more generic solution myself, but it would be cool if someone tackled it. :) Perhaps some of you also have ideas for other options for how to deal with this...

1 comment

heads-up: watch out for dependencies on "Plone"

by David Glick posted Mar 06, 2009 08:40 AM

Watch out if you're installing an add-on product to a Plone 3.1.x site that declares a dependency on "Plone" in its Plone wasn't packaged as an egg until Plone 3.2.x, so this will result in buildout trying to pull in the latest Plone on top of the one you've already got via the plone.recipe.plone recipe, and you'll see a VersionConflictError.

I just helped someone on #plone who ran into this with the collective.flowplayer product. If you're sure the product is actually compatible with Plone 3.1.x, you can work around this by adding "Plone" to the additional-fake-eggs option of the plone.recipe.zope2install buildout section.


report from the Berlinale sprint: how we can fix the Plone skin situation

by David Glick posted Feb 23, 2009 03:57 AM

Well! I am back from two wonderful weeks of traveling in Germany, Belgium, and Spain, and it's high time that I write something about what I got done at the Berlinale sprint which was a highlight of the trip and is already almost two weeks past!

I headed into the sprint with the goal of figuring out how to make Plone 4 not have two different ways of registering templates and resources, like we have in Plone 3. By the end of the week I had written only a little bit of code toward this end, but I was able to spend a lot of time talking to Hanno Schlichting, Laurence Rowe, and Andi Zeidler and figuring out how we'd like things to work. I feel quite good about the progress we were able to make toward a clearer vision of "what good looks like" in this area, and have a better idea now about what further code needs to be written to make that vision a reality.

To summarize our overall conclusion... I've heard a number of people say "skin layers should die." But this is a bit shortsighted, and probably reflects more the fact that people are tired of having two different ways to do this stuff, and that Zope 3 views and resources are a newer technology, rather than any clear superiority in functionality of the Zope 3 approaches over the alternative. So instead of ditching portal_skins, we should try to modernize it and make it compatible with the Zope 3 approaches.

But, this is a complicated thing to get right, and I would like to have feedback on whether we are on the right track and considering all the important use cases, so let me go into some detail on our current challenge and some concrete steps we can take to improve the situation.

The status quo

Currently in Plone 3 we have two different ways to register a resource that will be accessible via a URL in Zope. There are Content Management Framework (CMF) skin layers, which let you create a directory with a bunch of files (such as templates, images, Python scripts, CMF Form Controller objects, etc.) and associate it with a particular "theme" which may be enabled in your Plone site. These files are then magically available via Python attribute access on the portal root, thanks to a magic __getattr__ method in CMF's Skinnable class, which the portal root class extends.

There are also browser views and browser resources, concepts borrowed from Zope 3. These are generally registered via declarations in the XML-based Zope Configuration Markup Language (ZCML) which result in these resources getting registered as multi-adapters of a context and a request in the global registry (a.k.a. global site manager) of the Zope Component Architecture (ZCA). These adapters are then looked up as one of the steps in the traversal process of the Zope publisher.

Of course, it's confusing to have two different ways of solving the same problem that are both used in different places. (Just try to explain to a Plone newbie what the difference between portal_skins and portal_view_customizations is and when each should be used. There are answers, but they depend mostly on the technical details of the implementation and the history of Plone's development, and are not much use to someone actually trying to use these systems to solve the practical question of how to override component X.) We'd really like to have one system for registering resources. But the problem is that neither of the existing approaches is an obvious winner. Each one has benefits and tradeoffs.

Comparison of CMF skin layers vs. Zope 3 views and resources

A. CMF skin layers / portal_skins
Pros Cons
  • Very easy both to add a new resource and to override an existing one
  • The concept of overriding a resource through the web by copying it to a 'custom' layer is fairly intuitive
  • Supports TTW customization for many different types of resources (templates, images, CSS, JS, KSS, etc.)
  • Includes a diff feature for comparing customized items to the filesystem version.
  • Includes a Find feature for determining which version of an item is active.
  • Tools exist to aid with exporting TTW customizations back to the filesystem.
  • Supports the concept of multiple themes which may be activated at different times
  • Resources are available globally and can collide with content ids
  • The way resources are traversed is a bit magical and may slow attribute access for anything acquired from the portal
  • A bit cumbersome to register a new layer (requires ZCML and Generic Setup)

B. Zope 3 views & resources / portal_view_customizations
Pros Cons
  • Views and resources can be registered more selectively for a particular interface only, so that two resources of the same name can be used in different contexts.
  • Built on infrastructure that is more "core" to Zope (and thus probably better maintained and optimized)
  • Resources can be namespaced by using resource directories, and thereby separated from content.
  • The portal_view_customizations UI is rather poor and doesn't support some of the nice features of portal_skins.
  • Adding a new view or overriding an existing one requires a new ZCML registration.
  • There's no way to override just one item from a resource directory.
  • Through-the-web overrides are only supported for templates.
  • There's no good way to export through-the-web customizations back to the filesystem.
  • No simple way to handle subsite theming...a number of people have been struggling with this lately.

So what do we do?

So how can we continue our modernization toward Zope 3 approaches, without losing all the good things about CMF skin layers?

Based on the conversations I had at the Berlinale sprint, I think the answer involves three parts...

1. Register skin layer items as Zope 3 views and resources

The big coding task I tackled at the sprint was creating a package called 'plone.resource'. This is intended to be a reimplementation of the DirectoryInformation class from CMF (the class which handles exposing filesystem items as objects within Zope) but with different behavior behind the scenes, such that each item in the directory gets registered as a browser view.

Some more details:

  • It will be possible to register new resources (including template-based browser views) just by creating a file in the right directory. If Zope is running in debug mode, we can detect new and removed files without needing a restart.
  • All the items in a particular resource directory will be registered for a particular, configurable Zope 3 browser layer.
  • For templates, additional parameters to the view registration (such as "for" and "name") can be specified either in .metadata files or in <meta> tags in the <head> of the template's HTML. (Of course, ZCML registrations for browser views not located in a plone.resource-managed directory will continue to work.)
  • It will be possible to configure the resources associated with a directory to show up in the global namespace a la portal_skins (for cases where you really do want a resource accessible anywhere), in a ++resource++ namespace directory a la Zope 3 resources (mostly for backwards compatibility), or in a named path like 'images/' (for cases where you want resources separated from content, but want a prettier URL than ++resource++).
  • Because everything gets registered as a browser view in the end, it will be easier for us to create a single UI which can handle customizing all kinds of resources.
  • It should be possible to override existing resources from another package in a fashion similar to z3c.jbot, by creating a file whose name is the full dotted path to the resource being overridden.
  • For now I am reusing the resource classes from CMF; though, as Hanno pointed out, it may be possible to replace these with something more lightweight at some point. Supporting DTML, CMFFormController, and Python scripts in the long term are not priorities for me, although I hope to make plone.resource pluggable enough that support for them and other types of resources could be provided by add-on products.

A lot of this still needs to be built, but the foundation is already in place.

2. Improve our mechanisms for managing how browser layers get activated

Since more things will depend on Zope 3 browser layers, we will need better support in plone.browserlayer for:

  • Making sure that items on a "theme" layer take precedence over items on an "add-on product" layer. (That is, controlling the order in which layer interfaces are applied to the request).
  • Enabling a theme layer only for certain sections of the site (as well as disabling any other theme layer within that section).

I haven't done much thinking yet about the details of either the UI or the implementation for these enhancements. Suggestions welcome.

3. Revamp the UI for customization

portal_view_customizations is functional, but it needs a new user interface...possibly one that is based in the Plone control panel rather than the ZMI. I envision a listing of views and resources driven by a form that lets you filter by name of resource, type of resource, layer it's registered on, interface that a view is "for", etc. (Ideally we could even support a full text search of template contents.) For each resource shown in the results listing, all of the layers providing that resource need to be made evident, along with an indication of which is currently taking precedence. And it should be possible to see a diff of changes for items that have been customized through the web, as well as to export all such items for use in filesystem resource directories.

Alternatively, or in addition, an approach more like gloworm (which operates by the "poke-that-thing-on-the-page-and-tell-me-more-about-it-style" interface) could be taken. As long as plone.resource (or whatever else registers views) takes advantage of zope.configuration's support for annotating the view registration with info about where the registration originated, and as long as we have some way to expose this when a view's template is rendered, this shouldn't be too hard to achieve.

Aside from the new things that need to be built, the main implications this has for the existing implementation are:

  • We need to support other types of resources besides templates, both in terms of finding them and storing persistent copies of them. (I need to investigate further whether it's possible to adapterize the function of getting file contents somehow, so that we don't need to have separate persistent and non-persistent versions of each resource class.)
  • When an item gets customized through the web and makes a local copy of the view registration, we need to change the layer to something like ITTWCustomized so that it doesn't mask the global registration for the purposes of introspection.

A few last words

Okay, I think that's probably enough to keep me busy for a little while. :) Please don't hesitate to ask questions; this plan incorporates a lot of small improvements and there are no doubt places where I should have been clearer about what something means or why something is a good idea. Constructive criticism and suggestions on how to make this even better are also welcome.

You'll no doubt notice that I haven't said much (if anything) about viewlets and portlets. Don't worry, we know that understanding the distinction between these and learning how to create and register them is one of the common pain points in learning Plone theming today, and we hope to make things easier in Plone 4. But this problem is mostly orthogonal to the question of how resources get registered and customized, and is being dealt with elsewhere, so I'm not going to spend time on it here. For now, suffice it to say that as long as the approach used in Plone 4 for generating a section of a page is based on a Zope 3 browser view, it should be compatible with the improvements discussed here.

David Glick

David Glick

I am a problem solver trying to make websites easier to build.

Currently I do this in my spare time as a member of the Plone core team, and during the day as an independent web developer specializing in Plone and custom Python web applications.