Skip to content. | Skip to navigation

Personal tools

>>> ''.join(word[:3].lower() for word in 'David Isaac Glick'.split())

‘davisagli’

Navigation

You are here: Home

David Glick – Plone developer

by admin posted Apr 06, 2010 02:48 AM

Be vigilant: Know your Buildout Threat Level

by David Glick posted May 10, 2010 12:09 AM

The Buildout Threat Level indicator reports the percentage of buildouts that have succeeded in the past 4 hours.

For those who missed my announcement last week on Twitter –

As a result of a mostly tongue-in-cheek conversation on IRC with Elizabeth Leddy, Alex Clark, and Matthew Wilkes, I decided to set up the global zc.buildout threat level indicator.

The indicator collects reports of success and failure from actual buildout runs, and displays the current threat level based on the percentage of buildouts that have succeeded in the past 4 hours.

I'm happy to report that the current threat level is LOW.

Buildout Threat Level: Low

Creating the indicator was mostly an excuse to try writing a Google App Engine app, but it's already proved its utility.  Last Thursday the threat level reached ELEVATED after a new distribute release started causing buildouts to fail under Python 2.4. Thanks to Tarek Ziadé for the quick response with a new distribute release to fix that.

You can help make the indicator more accurate!  Just add the buildout.threatlevel extension to your buildout:

[buildout]
extensions = buildout.threatlevel

This will display the current threat level as one of the initial steps when you run buildout (so that you have an opportunity to abort if it has reached SEVERE), and will ping my app engine app when the buildout is done to report success or failure. (No data is collected in this ping aside from a boolean indicating success, time, and IP address.)

Code for the GAE app and buildout extension is in the collective.

4 comments

Report-out from the Seattle GetPaid recurring payment mini-sprint

by David Glick posted May 09, 2010 11:25 PM

Six Plone developers in Seattle gathered this past weekend to work on adding support for recurring payments to PloneGetPaid.

Groundwire organized a mini-sprint this past Thursday-Saturday to work on adding support for handling recurring subscription-based payments using PloneGetPaid.  I had the honor of coordinating this work, and am happy to report that we had a very productive sprint and surpassed my goals for what we should accomplish.

We added support for creating a recurring payment subscription to the Authorize.net and Paypal payment processors.  This will be done instead of a standard one-time payment if the cart contains a "recurring line item," with a corresponding interval and total # of payments.  (Carts containing multiple recurring items, or a mixture of recurring and non-recurring items, are not supported.)

We also created two ways to get recurring line items into the cart.  The first way is similar to the way that other types of payment are configured in GetPaid: Add a marker to an arbitrary piece of content by selecting "Make recurring payment" from the action menu, and then enter the price, interval, and total # of payments.  (This can be done as long as the current payment processor supports recurring payments.)  Once an item is configured as a recurring payment, a portlet displays with an "Add to Cart" button that can be used to being purchasing.

The second way to get recurring line items into the cart is via a new package, pfg.donationform, which provides a shortcut to set up a PloneFormGen form for making a one-time *or* recurring donation.  (Aside from the recurrence part this was possible before using getpaid.formgen, but the new package streamlines the setup.)  Selecting "Donation Form" from Plone's Add menu gives the site manager an opportunity to enter several settings which affect the creation of the standard PloneFormGen form.  The created form includes a custom "Donation Fieldset" which allows selecting from predefined donation levels or entering a custom amount, and specifying whether the payment should be one-time or recurring, and for how many months it should recur.  The donation form may also optionally be populated with fields to collect contact and billing info so that checkout can happen in a single step (at least with an on-site payment processor like the Authorize.net one).  When the donation form is submitted, line items will be added to the GetPaid cart based on the settings in the Donation Fieldset, and checkout will be initiated.

pfg.donationform

Along the way, we also found time to add tests to getpaid.authorizedotnet and getpaid.formgen, which had no working tests before.

All of this work has been on branches so far.  In the next week or so we'll be working on adding a few more tests, merging the changes, and making new releases of the relevant packages.  If you would like to review the work before that happens, please feel free.  Just check out the getpaid development buildout, and build it using recurring-payment.cfg, to get the relevant branches.

Thanks to the other sprinters and their employers for giving your time: Alex Tokar and Fulvio Casali of Web Collective, Jesse Snyder of NPower Seattle, Cris Ewing of U. of Washington Radiology, and Rob Larubbio.  (And welcome aboard as GetPaid committers, Alex, Fulvio, Jesse, and Cris.)  Thanks also to ifPeople and Juan Pablo Gímenez who did some work in 2008 that provided a helpful starting point for our work, to Zvezdan Petkovic from Zope Corporation for making new Zope 2.12-compatible releases of zc.ssl and zc.authorizedotnet, and to Groundwire for giving us a space to work and buying us lunch on Friday.

Organizing a mini-sprint

Some scattered thoughts on organizing an effective sprint:

  • Sprints don't have to be large, expensive, or difficult to organize. If you have a few local Plonistas, a place to meet, and a clear shared goal, you can have a great sprint.  A small sprint means fewer brains, but increased communication and focus.
  • Not everyone participating in the sprint needs to have a deep familiarity with the product you're working on, but if you want to maximize productivity, everyone should have prior experience working on some Plone product.  And at least one person needs to have enough familiarity with the product and the sprint goal to produce a list of tasks ahead of time that can be easily divvied up amongst the participants.
  • A small amount of effort by the sprint coordinator in advance can make sure half the sprint isn't wasted on initial setup and bootstrapping tasks that don't benefit from more people.  Prior to the sprint, I made sure that Juan's previous branches were up-to-date against GetPaid trunk with passing tests (well, mostly).  I also made sure that all participants had commit access to the GetPaid repository, and tried to make sure that they ran the development buildout ahead of time.
3 comments

Notes on migrating this blog from Wordpress to Plone

by David Glick posted Apr 11, 2010 05:06 AM

Includes example code for exporting blog posts and comments from Wordpress and importing them into Plone.

I'm proud to be now running this blog on Plone 4, having just completed a migration from Wordpress.  Here are a few notes on the migration.

Initial steps that I won't go into detail on, at least for now:

  • Obtained a Linode VPS with enough RAM to feed Plone (it's looking like my small Plone 4 site will need about 100MB to itself ... a greedy baseline, but that does include the database).
  • Created a Plone 4 buildout with ZEO and one Zope instance being run under supervisord.
  • Installed plone.app.caching, the new caching framework Martin Aspeli and Ric Newbery have been working on.  This still requires a number of svn checkouts at this point, but it's coming along nicely.
  • Installed Scrawl so that I can manage blog entries as a separate content type.
  • Installed plone.app.discussion, the new comment and discussion add-on created by Timo Stollenwerk, and its recaptcha add-on.  Made sure comments were enabled for the Blog Entry type, and that comment moderation was not turned on for the duration of the migration.
  • Turned off much of Plone's default HTML filtering.  Security is not a huge concern since I'm going to be the only one editing the site, and I tend to want to use fancy stuff in posts sometimes.

Data Export

Moving the posts and comments from my old blog's MySQL database was easier than I feared, though I did have to do a bit of coding.

I decided up front to do this by way of dumping the data to CSV files, rather than writing import code that read directly from MySQL. That was mostly a visceral reaction to a memory of a hard time getting MySQL-python installed and working properly once previously, and may have been an irrational fear.  But dumping the data from MySQL to CSV was easy enough, with the following two queries that grabbed just the data I needed:

SELECT ID, post_date, CONVERT(post_content USING latin1), post_title
INTO OUTFILE '/tmp/musings.csv'
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '\\' LINES TERMINATED BY '\n'
FROM wp_posts, wp_post2cat
WHERE wp_posts.ID=post_id AND category_id=48
AND post_status='publish';

SELECT comment_post_ID, comment_author, comment_author_email, comment_author_url, comment_date, CONVERT(comment_content USING latin1), user_id
INTO OUTFILE '/tmp/comments.csv'
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '\\' LINES TERMINATED BY '\n'
FROM wp_comments
WHERE comment_approved = '1' AND comment_type='';

This grabs a bunch of fields and dumps them into a CSV file with the given CSV dialect parameters.  The only tricky bit here is the call to CONVERT, which was needed because my raw data in MySQL had been improperly encoded.  A normal connection to MySQL defaults to the latin1 encoding (which is what MySQL calls windows-1252).  But Wordpress had been sending it utf8-encoded data, and the MySQL table had been configured to store things as utf8.  So when I stored data, MySQL was decoding the utf8 input as windows-1252, and then re-encoding as utf8.  On retrieval via a normal connection the reverse transformation was applied and it didn't matter, but SELECT INTO OUTFILE just copies the raw data from the table, which was effectively gobbledygook.  So I had to explicitly make MySQL convert the stored value to latin1 (read: decode it as utf8 and then encode as windows-1252) in order to end up with the utf8 I wanted.  This would have been needed for the other fields as well, except I wasn't using non-ASCII characters in them.

The category restriction on the first query makes sure that I only got the Plone-related posts from the old blog (I had been using it for personal blogging as well).  The comment type restriction on the second query excludes pingbacks.

Data Import

I ended up writing this custom External Method to import the CSV data into Plone.  (I looked at transmogrifier, csvreplicata, and ArcheCSV, but for all of these it looked like I would have ended up writing a significant amount of code in the end anyway to get them to do what I wanted. And I knew I could do that "from scratch" in just a page or two of Python...)

import csv
import re
from DateTime import DateTime
from zope.component import queryUtility, createObject
from plone.i18n.normalizer.interfaces import IIDNormalizer
from plone.app.discussion.interfaces import IConversation

PRE_RE = re.compile(r'(<pre>.*?</pre>)', re.IGNORECASE | re.DOTALL)

def cleanup_wordpress_text(text):
    text = PRE_RE.sub(lambda x: x.group(1).replace('\r\n\r\n', '\n\n'), text)
    return text.replace('\r\n\r\n', '<p>').replace('\r\n','\n').decode('utf-8')

def importmusings(self):
    context = self
    reader = csv.reader(open('/tmp/musings.csv'), delimiter=',', quotechar='"', doublequote=False, escapechar='\\')
    posts = {}
    for row in reader:
        id, date, text, title = row
        short = queryUtility(IIDNormalizer).normalize(title)
        if short in context:
            del context[short]
        post = context[context.invokeFactory('Blog Entry', short)]
        post.setCreators(['davisagli'])
        post.setEffectiveDate(DateTime(date))
        post.setTitle(title)
        text = cleanup_wordpress_text(text)
        post.setText(text, mimetype='text/html')
        post.reindexObject()
        context.portal_workflow.doActionFor(post, 'publish')
        posts[id] = post

    reader = csv.reader(open('/tmp/comments.csv'), delimiter=',', quotechar='"', doublequote=False, escapechar='\\')
    for row in reader:
        post_id, author, email, url, date, text, uid = row
        try:
            post = posts[post_id]
        except KeyError:
            continue
        conversation = IConversation(post)
        comment = createObject('plone.Comment')
        comment.text = cleanup_wordpress_text(text)
        if uid == '1':
            comment.creator = comment.author_username = 'davisagli'
            comment.author_name = 'David Glick'
            comment.author_email = 'dglick@gmail.com'
        else:
            comment.creator = None
            comment.author_name = author
            comment.author_email = email
        date = DateTime(date).asdatetime()
        comment.creation_date = comment.modification_date = date
        conversation.addComment(comment)
 
    return 'Done.'

That cleanup_wordpress_text function turns double newlines from Wordpress into proper paragraph tags -- unless they're within a PRE tag.  The rest of the code is pretty readable -- yay, Python.

Syntax Highlighting

You probably noticed one of the new site features -- syntax highlighting for blocks of code.  This is provided by the Pygments module, applied as a transformation to the entire response just before the Zope publisher returns it.  I achieved that via a plugin (called collective.pygmentstransform, available in the collective, and not released so far or probably ever) for Martin's plone.transformchain (also unreleased so far).  It's imperfect (not least because it guesses the language heuristically), but good enough for now I think.  Yes, I should probably be doing this as WSGI middleware, but I haven't spent the time to figure out how to run Zope 2.12 under WSGI yet.

7 comments
David Glick

David Glick

I am a problem solver trying to make websites easier to build.

Currently I do this in my spare time as a member of the Plone core team, and during the day as an independent web developer specializing in Plone and custom Python web applications.