Skip to content. | Skip to navigation

Personal tools

>>> ''.join(word[:3].lower() for word in 'David Isaac Glick'.split())

‘davisagli’

Navigation

You are here: Home

David Glick – Plone developer

by admin posted Apr 05, 2010 11:48 PM

Report-out from the Seattle GetPaid recurring payment mini-sprint

by David Glick posted May 09, 2010 08:25 PM

Six Plone developers in Seattle gathered this past weekend to work on adding support for recurring payments to PloneGetPaid.

Groundwire organized a mini-sprint this past Thursday-Saturday to work on adding support for handling recurring subscription-based payments using PloneGetPaid.  I had the honor of coordinating this work, and am happy to report that we had a very productive sprint and surpassed my goals for what we should accomplish.

We added support for creating a recurring payment subscription to the Authorize.net and Paypal payment processors.  This will be done instead of a standard one-time payment if the cart contains a "recurring line item," with a corresponding interval and total # of payments.  (Carts containing multiple recurring items, or a mixture of recurring and non-recurring items, are not supported.)

We also created two ways to get recurring line items into the cart.  The first way is similar to the way that other types of payment are configured in GetPaid: Add a marker to an arbitrary piece of content by selecting "Make recurring payment" from the action menu, and then enter the price, interval, and total # of payments.  (This can be done as long as the current payment processor supports recurring payments.)  Once an item is configured as a recurring payment, a portlet displays with an "Add to Cart" button that can be used to being purchasing.

The second way to get recurring line items into the cart is via a new package, pfg.donationform, which provides a shortcut to set up a PloneFormGen form for making a one-time *or* recurring donation.  (Aside from the recurrence part this was possible before using getpaid.formgen, but the new package streamlines the setup.)  Selecting "Donation Form" from Plone's Add menu gives the site manager an opportunity to enter several settings which affect the creation of the standard PloneFormGen form.  The created form includes a custom "Donation Fieldset" which allows selecting from predefined donation levels or entering a custom amount, and specifying whether the payment should be one-time or recurring, and for how many months it should recur.  The donation form may also optionally be populated with fields to collect contact and billing info so that checkout can happen in a single step (at least with an on-site payment processor like the Authorize.net one).  When the donation form is submitted, line items will be added to the GetPaid cart based on the settings in the Donation Fieldset, and checkout will be initiated.

pfg.donationform

Along the way, we also found time to add tests to getpaid.authorizedotnet and getpaid.formgen, which had no working tests before.

All of this work has been on branches so far.  In the next week or so we'll be working on adding a few more tests, merging the changes, and making new releases of the relevant packages.  If you would like to review the work before that happens, please feel free.  Just check out the getpaid development buildout, and build it using recurring-payment.cfg, to get the relevant branches.

Thanks to the other sprinters and their employers for giving your time: Alex Tokar and Fulvio Casali of Web Collective, Jesse Snyder of NPower Seattle, Cris Ewing of U. of Washington Radiology, and Rob Larubbio.  (And welcome aboard as GetPaid committers, Alex, Fulvio, Jesse, and Cris.)  Thanks also to ifPeople and Juan Pablo Gímenez who did some work in 2008 that provided a helpful starting point for our work, to Zvezdan Petkovic from Zope Corporation for making new Zope 2.12-compatible releases of zc.ssl and zc.authorizedotnet, and to Groundwire for giving us a space to work and buying us lunch on Friday.

Organizing a mini-sprint

Some scattered thoughts on organizing an effective sprint:

  • Sprints don't have to be large, expensive, or difficult to organize. If you have a few local Plonistas, a place to meet, and a clear shared goal, you can have a great sprint.  A small sprint means fewer brains, but increased communication and focus.
  • Not everyone participating in the sprint needs to have a deep familiarity with the product you're working on, but if you want to maximize productivity, everyone should have prior experience working on some Plone product.  And at least one person needs to have enough familiarity with the product and the sprint goal to produce a list of tasks ahead of time that can be easily divvied up amongst the participants.
  • A small amount of effort by the sprint coordinator in advance can make sure half the sprint isn't wasted on initial setup and bootstrapping tasks that don't benefit from more people.  Prior to the sprint, I made sure that Juan's previous branches were up-to-date against GetPaid trunk with passing tests (well, mostly).  I also made sure that all participants had commit access to the GetPaid repository, and tried to make sure that they ran the development buildout ahead of time.
3 comments

Notes on migrating this blog from Wordpress to Plone

by David Glick posted Apr 11, 2010 02:06 AM

Includes example code for exporting blog posts and comments from Wordpress and importing them into Plone.

I'm proud to be now running this blog on Plone 4, having just completed a migration from Wordpress.  Here are a few notes on the migration.

Initial steps that I won't go into detail on, at least for now:

  • Obtained a Linode VPS with enough RAM to feed Plone (it's looking like my small Plone 4 site will need about 100MB to itself ... a greedy baseline, but that does include the database).
  • Created a Plone 4 buildout with ZEO and one Zope instance being run under supervisord.
  • Installed plone.app.caching, the new caching framework Martin Aspeli and Ric Newbery have been working on.  This still requires a number of svn checkouts at this point, but it's coming along nicely.
  • Installed Scrawl so that I can manage blog entries as a separate content type.
  • Installed plone.app.discussion, the new comment and discussion add-on created by Timo Stollenwerk, and its recaptcha add-on.  Made sure comments were enabled for the Blog Entry type, and that comment moderation was not turned on for the duration of the migration.
  • Turned off much of Plone's default HTML filtering.  Security is not a huge concern since I'm going to be the only one editing the site, and I tend to want to use fancy stuff in posts sometimes.

Data Export

Moving the posts and comments from my old blog's MySQL database was easier than I feared, though I did have to do a bit of coding.

I decided up front to do this by way of dumping the data to CSV files, rather than writing import code that read directly from MySQL. That was mostly a visceral reaction to a memory of a hard time getting MySQL-python installed and working properly once previously, and may have been an irrational fear.  But dumping the data from MySQL to CSV was easy enough, with the following two queries that grabbed just the data I needed:

SELECT ID, post_date, CONVERT(post_content USING latin1), post_title
INTO OUTFILE '/tmp/musings.csv'
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '\\' LINES TERMINATED BY '\n'
FROM wp_posts, wp_post2cat
WHERE wp_posts.ID=post_id AND category_id=48
AND post_status='publish';

SELECT comment_post_ID, comment_author, comment_author_email, comment_author_url, comment_date, CONVERT(comment_content USING latin1), user_id
INTO OUTFILE '/tmp/comments.csv'
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '\\' LINES TERMINATED BY '\n'
FROM wp_comments
WHERE comment_approved = '1' AND comment_type='';

This grabs a bunch of fields and dumps them into a CSV file with the given CSV dialect parameters.  The only tricky bit here is the call to CONVERT, which was needed because my raw data in MySQL had been improperly encoded.  A normal connection to MySQL defaults to the latin1 encoding (which is what MySQL calls windows-1252).  But Wordpress had been sending it utf8-encoded data, and the MySQL table had been configured to store things as utf8.  So when I stored data, MySQL was decoding the utf8 input as windows-1252, and then re-encoding as utf8.  On retrieval via a normal connection the reverse transformation was applied and it didn't matter, but SELECT INTO OUTFILE just copies the raw data from the table, which was effectively gobbledygook.  So I had to explicitly make MySQL convert the stored value to latin1 (read: decode it as utf8 and then encode as windows-1252) in order to end up with the utf8 I wanted.  This would have been needed for the other fields as well, except I wasn't using non-ASCII characters in them.

The category restriction on the first query makes sure that I only got the Plone-related posts from the old blog (I had been using it for personal blogging as well).  The comment type restriction on the second query excludes pingbacks.

Data Import

I ended up writing this custom External Method to import the CSV data into Plone.  (I looked at transmogrifier, csvreplicata, and ArcheCSV, but for all of these it looked like I would have ended up writing a significant amount of code in the end anyway to get them to do what I wanted. And I knew I could do that "from scratch" in just a page or two of Python...)

import csv
import re
from DateTime import DateTime
from zope.component import queryUtility, createObject
from plone.i18n.normalizer.interfaces import IIDNormalizer
from plone.app.discussion.interfaces import IConversation

PRE_RE = re.compile(r'(<pre>.*?</pre>)', re.IGNORECASE | re.DOTALL)

def cleanup_wordpress_text(text):
    text = PRE_RE.sub(lambda x: x.group(1).replace('\r\n\r\n', '\n\n'), text)
    return text.replace('\r\n\r\n', '<p>').replace('\r\n','\n').decode('utf-8')

def importmusings(self):
    context = self
    reader = csv.reader(open('/tmp/musings.csv'), delimiter=',', quotechar='"', doublequote=False, escapechar='\\')
    posts = {}
    for row in reader:
        id, date, text, title = row
        short = queryUtility(IIDNormalizer).normalize(title)
        if short in context:
            del context[short]
        post = context[context.invokeFactory('Blog Entry', short)]
        post.setCreators(['davisagli'])
        post.setEffectiveDate(DateTime(date))
        post.setTitle(title)
        text = cleanup_wordpress_text(text)
        post.setText(text, mimetype='text/html')
        post.reindexObject()
        context.portal_workflow.doActionFor(post, 'publish')
        posts[id] = post

    reader = csv.reader(open('/tmp/comments.csv'), delimiter=',', quotechar='"', doublequote=False, escapechar='\\')
    for row in reader:
        post_id, author, email, url, date, text, uid = row
        try:
            post = posts[post_id]
        except KeyError:
            continue
        conversation = IConversation(post)
        comment = createObject('plone.Comment')
        comment.text = cleanup_wordpress_text(text)
        if uid == '1':
            comment.creator = comment.author_username = 'davisagli'
            comment.author_name = 'David Glick'
            comment.author_email = 'dglick@gmail.com'
        else:
            comment.creator = None
            comment.author_name = author
            comment.author_email = email
        date = DateTime(date).asdatetime()
        comment.creation_date = comment.modification_date = date
        conversation.addComment(comment)
 
    return 'Done.'

That cleanup_wordpress_text function turns double newlines from Wordpress into proper paragraph tags -- unless they're within a PRE tag.  The rest of the code is pretty readable -- yay, Python.

Syntax Highlighting

You probably noticed one of the new site features -- syntax highlighting for blocks of code.  This is provided by the Pygments module, applied as a transformation to the entire response just before the Zope publisher returns it.  I achieved that via a plugin (called collective.pygmentstransform, available in the collective, and not released so far or probably ever) for Martin's plone.transformchain (also unreleased so far).  It's imperfect (not least because it guesses the language heuristically), but good enough for now I think.  Yes, I should probably be doing this as WSGI middleware, but I haven't spent the time to figure out how to run Zope 2.12 under WSGI yet.

7 comments

Tahoe sprint off and running

by David Glick posted Mar 16, 2010 01:59 AM

After carpooling together from San Francisco yesterday afternoon, the five participants in the Tahoe Snow Sprint arrived at our swanky lodgings overlooking Lake Tahoe. The decor may be questionable, but there is more than enough space for us all, leaving us with crucial questions such whether to use the first floor or third floor bar at any given time.

We haven't done much coding yet, as much of the evening was spent with me giving a walkthrough of current Dexterity functionality, accompanied by much discussion of what works well and what doesn't, and what things we'd like to work on improving.

I will work on making various improvements to the usability and functionality of the through-the-web content type editor, hopefully with some help on UI design from Alex Limi.

Alex also hopes to work on designing some improved widgets.

Ross Patterson will work on adding support for Choice fields with vocabularies and sources to the TTW editor.

David Brenneman will look into ways of allowing Archetypes content to reference Dexterity content.

And I believe Joel Burton will work on exporting content types created through the web to a full installable package (as opposed to simply exporting the FTI like you can do already via portal_setup).

So let the sprinting begin!

We are hanging out in #sprint on freenode, so feel free to stop by and say hello to the tahoebot.

David Glick

David Glick

I am a problem solver trying to make websites easier to build.

Currently I do this in my spare time as a member of the Plone core team, and during the day as an independent web developer specializing in Plone and custom Python web applications.