Skip to content. | Skip to navigation

Personal tools

>>> ''.join(word[:3].lower() for word in 'David Isaac Glick'.split())

‘davisagli’

Navigation

You are here: Home

David Glick – Plone developer

by admin posted Apr 05, 2010 11:48 PM

Visualizing the ZODB with graphviz

by David Glick posted May 22, 2009 11:25 AM

While digging around in the ZEXP export code, I realized that it wouldn't be too hard to modify it to dump a representation of a ZODB in graphviz .dot format. Here's a Zope external method I devised to do that:

 

# Generic ZODB walker and graphviz exporter

####################################################################
#
# Copyright (c) 2003 Zope Corporation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.1 (ZPL).  A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE.
#
####################################################################

import logging
import cPickle, cStringIO
from ZODB.utils import u64

logger = logging.getLogger('ZODB.ExportImport')

def get_reference_dumper(refs):
    # This is a callback which will be called whenever a reference is found.
    def dump_reference(oid, roid):
        refs.append('%s -> %s\n' % (u64(oid), u64(roid)))
    return dump_reference

def export_graphviz(self):
    """
    Walks a ZODB database and dumps the object graph in graphviz .dot format.
    """
    context = self
    f = open('plone.dot', 'w')
    f.write('digraph plone {\n')
    refs = []
    reference_dumper = get_reference_dumper(refs)
    for oid, p in walk_database(context, reference_callback=reference_dumper):
        # Walk to all the objects in the database and examine their references.
        # Whenever a reference is found, it will be recorded via the
        # reference_dumper.  Whenever a new object is found, it will be yieled
        # to this loop.

        # Read the module and class from the pickle bytestream without actually
        # loading the object.
        module, klass = p.split('\n')[:2]
        module = module[2:]
        
        f.write('%s [label="%s.%s"]\n' % (u64(oid), module, klass))
    for ref in refs:
        f.write(ref)
    f.write('}\n')
    f.close()

def walk_database(context, reference_callback=None):
    # Get the object ID and database connection of the starting object.
    base_oid = context._p_oid
    conn = context._p_jar
    
    # oids is used to keep track of found oids that need to be visited.
    # done_oids is used to keep track of which oids have already been yielded.
    oids = [base_oid]
    done_oids = {}
    while oids:
        # loop while references remain to objects we haven't exported yet
        oid = oids.pop(0)
        if oid in done_oids:
            continue
        done_oids[oid] = True
        
        try:
            # fetch the pickle
            p, serial = conn._storage.load(oid, conn._version)
        except:
            logger.debug("broken reference for oid %s", repr(oid),
                         exc_info=True)
        else:
            # If the Unpickler's persistent_load attribute is set to a list,
            # then that list will be populated with the references found in
            # the pickle when noload is called, without actually loading the
            # object.
            refs = []
            u = cPickle.Unpickler(cStringIO.StringIO(p))
            u.persistent_load = refs
            # noload must be called the same # of times it was called when
            # pickling
            u.noload()
            u.noload()

            # loop through the references found on this object
            for ref in refs:

                # look for the various reference types supported by the ZODB
                # (see the docs in ZODB/serialize.py for details)
                if isinstance(ref, tuple):
                    roid = ref[0]
                elif isinstance(ref, str):
                    roid = ref
                else:
                    try:
                        ref_type, args = ref
                    except ValueError:
                        # weakref - not supported
                        continue
                    else:
                        if ref_type in ('m', 'n'):
                            # cross-database ref - not supported
                            continue
                if roid:
                    # record this reference
                    if reference_callback:
                        reference_callback(oid, roid)

                    # add the referenced object to the list of objects we need
                    # to visit
                    oids.append(roid)

            # yield the oid and pickle
            yield oid, p

Download graphviz_export.py

And after running this on a fresh Plone site, sending the result through dot and loading it in zgrviewer, here's the result:

ZODB graphviz visualization

The site root is toward the upper right; most of the graph is persistent tools and such rather than actual content, since there is minimal content in a fresh Plone installation. That hairy mess on the left is the mimetype registry. Any resemblance to the shape of the BFG logo is entirely coincidental.

I'm not really sure what sort of useful information one might be able to get using this sort of technique, but I'm sure there are some possibilities, so please let me know if you have ideas or if you modify this to do something cool.

I want to try this on a site that has real data in it, but at the moment I'm waiting for the latest XCode to download so that I can build the newest graphviz which includes sfdp which is supposed to be better for handling really big graphs.

2 comments

New product: collective.weightedportlets

by David Glick posted May 01, 2009 04:45 AM

Ever been frustrated with not being able to control exactly what order your portlets show up in, if you've got inherited portlets, content type portlets, and group portlets? I just released collective.weightedportlets, which allows you to specify an integer weight for each portlet assignment, which will be taken into account in the final ordering. See the Plone.org product page for details.

1 comment

A buildout for Plone 2.0.5

by David Glick posted Apr 28, 2009 03:13 PM

At ONE/Northwest we're always looking for ways to improve and streamline our system administration tasks. Recently, we've been working on converting all our old Zope instances to be buildout-based (to make it easier to recreate the environment for local testing of changes or in case the instance needs to move to another server). Here are some tips based on things we've learned in the process of putting together our buildout for Plone 2.0.5 ...

Use the right Python

Plone 2.0.5 is based on Zope 2.7, which requires Python 2.3 rather than Python 2.4 like modern versions of Zope. (We tested using Python 2.4 and it seems to work okay; however Zope 2.7's RestrictedPython has not been audited in Python 2.4 and there's no guarantee that users with the rights to edit scripts won't be able to do something nasty.)

I installed Python 2.3 using macports, then made sure to bootstrap and run my buildout using Python 2.3. Buildout initially complained about the 'subprocess' module being missing, but I was able to work around this by copying subprocess.py from my Python 2.4 libs (/opt/local/lib/python2.4/subprocess.py in my case) into the Python 2.3 libs.

Update: Recent versions of plone.recipe.zope2install use some Python generators which aren't compatible with Python 2.3, so I had to pin this egg to version 3.2.

Fry up some products

In a classic Zope installation you keep all your products in one Products directory. In buildout they are typically spread between several different product directories. In the case of Zope 2.7, we were seeing an issue where it only found products located in a Products directory at the root of the buildout, even if we listed additional directories. To work around this, I used collective.recipe.omelette to symlink the various buildout-generated products dirs into the main Products dir that Zope finds. Always nice to find a new use for a tool I designed for a completely different problem!

We do our development on OS X which uses a case insensitive filesystem, so I started using /svnproducts as a replacement for the /products dir that is often found in buildouts, since this would otherwise conflict with the auto-generated /Products.

Your configuration is no good here

Unfortunately the zope.conf that the plone.recipe.zope2instance recipe generates contains a couple bits of configuration (verbose-security and default-zpublisher-encoding) that cause Zope 2.7 to barf, since they were not added until later versions of Zope. To work around this, we used sed (via the plone.recipe.command recipe) to remove the offending bits.

The buildout

I ripped out the bits specific to our own systems and ended up with the following, which incorporates the above learnings. If I didn't mess up while abridging it, it even works!

[buildout]
parts =
    plone
    zope2
    productdistros
    omelette
    instance
    fixer
versions = versions

[versions]
plone.recipe.zope2install = 3.2

[plone]
recipe = plone.recipe.distros
urls = http://heanet.dl.sourceforge.net/sourceforge/plone/Plone-2.0.5.tar.gz
nested-packages = Plone-2.0.5.tar.gz
version-suffix-packages = Plone-2.0.5.tar.gz

[zope2]
recipe = plone.recipe.zope2install
url = http://www.zope.org/Products/Zope/2.7.7/Zope-2.7.7-final.tgz
fake-zope-eggs = false

# Archetypes and kupu are not strictly required, but here's how to get them if you need them.
[productdistros]
recipe = plone.recipe.distros
urls =
    http://voxel.dl.sourceforge.net/sourceforge/archetypes/Archetypes-1.3.2-final-Bundle.tar.gz
    http://plone.org/products/kupu/releases/1.3.9/kupu.tgz
nested-packages =
    Archetypes-1.3.2-final-Bundle.tar.gz
version-suffix-packages =

[omelette]
recipe = collective.recipe.omelette
eggs =
packages =
    ${buildout:directory}/svnproducts .
    ${buildout:directory}/parts/productdistros .
    ${buildout:directory}/parts/plone .
location = ${buildout:directory}/Products

[instance]
recipe = plone.recipe.zope2instance
zope2-location = ${zope2:location}
user = admin:admin
http-address = 8080
debug-mode = on
verbose-security = on
products =
    ${buildout:directory}/Products

[fixer]
recipe = plone.recipe.command
command =   
    sed -i '' 's/verbose-security/#verbose-security/' ${buildout:directory}/parts/instance/etc/zope.conf
    sed -i '' 's/default-zpublisher-encoding/#default-zpublisher-encoding/' ${buildout:directory}/parts/instance/etc/zope.conf
update-command = ${fixer:command}

Many thanks to my colleague Jon Baldivieso who did some of the initial work on this buildout.

Update 5/1/2009: Added fake-zope-eggs = false to avoid trying to build fake eggs from a directory that doesn't exist in Zope 2.7.

Update 8/21/2009: Pinned plone.recipe.zope2install to version 3.2, as newer versions use Python generators that aren't compatible with Python 2.3.

2 comments
David Glick

David Glick

I am a problem solver trying to make websites easier to build.

Currently I do this in my spare time as a member of the Plone core team, and during the day as an independent web developer specializing in Plone and custom Python web applications.