Skip to content. | Skip to navigation

Personal tools

>>> ''.join(word[:3].lower() for word in 'David Isaac Glick'.split())

‘davisagli’

Navigation

You are here: Home

David Glick – Plone developer

by admin posted Apr 05, 2010 11:48 PM

Recent improvements to Plone-Salesforce integration

by David Glick posted Jul 10, 2013 11:45 PM

The simple-salesforce client and a celery-based message queue are keys to making Plone-Salesforce integration simpler and more robust.

I've been working on yet another big project that involves synchronizing data between Plone and Salesforce. As in past projects, Plone provides a powerful web content management platform that can support lots of editors with a number of different roles without paying for additional license fees, while Salesforce's superior reporting tools provide motivation for replicating data there as well. With help from Jason Lantz, I've been working on modernizing the toolset used to achieve this replication.

The improvements are on several fronts:

  1. Using the simple-salesforce library instead of beatbox and suds
  2. Handling incoming and outgoing data via a message queue
  3. Using Plone ids as a external id in Salesforce

Let me cover each of these in more detail.

simple-salesforce vs. beatbox and suds

Salesforce originally had only a SOAP API, and for a long time the clients available for interacting with Salesforce from Python only worked with the SOAP API. There was beatbox, which supported the generic Salesforce partner webservice pretty well but didn't handle custom webservices. And there was suds, which could parse WSDL for custom webservices and allow interacting with them. This ecosystem was annoying: you needed different libraries for different tasks, they both were as error-prone as most SOAP clients are, and you needed to update WSDL files whenever someone made a change to the webservice.

These days Salesforce has a REST API—both for generic operations and available from Apex for custom webservices—and Jason called my attention to the simple-salesforce library from the team at New Organizing Institute. I read the code and was impressed: it's only about 500 lines of code and I don't think I would do much differently if I were writing it from scratch. (I'd probably add some tests, but it is working okay for me in practice.) Using simple-salesforce, inserting a contact looks something like this:

from simple_salesforce import Salesforce
sf = Salesforce(
    username='SF_USERNAME', password='SF_PASSWORD',
    security_token='SF_TOKEN', sandbox=True)
sf.Contact.insert({
    'FirstName': 'David',
    'LastName': 'Glick',
    'Email': 'david@glicksoftware.com',
})

Handling data via a message queue

The first generation of Plone-Salesforce integration tools made synchronous calls out to Salesforce. This was the easiest thing to get working, but had some problems:

  • High latency. This was a problem both for user experience and because it tied up server threads and increased the likelihood of database conflict errors.
  • Tight coupling. If Salesforce was temporarily unreachable due to network issues or a maintenance window, the Plone user would see errors.

To solve these problems we knew we needed a message queue to allow for asynchronous processing of data being relayed from one system to the other. We did some research and chose celery (with RabbitMQ as a backend). Celery has a lot of features including support for multiple backends and the flower UI for viewing the queue, and it seems to have some traction in other parts of the Python web programming world.

At first I was worried that celery would be difficult to integrate with Zope, but it turns out it's not very complicated and hardly even warrants the creation of a new library. Celery uses a connection pool, so can be used from within a threaded environment like Zope. So the main consideration is making sure that it plays nicely with Zope's transaction management: you don't want a job to get queued twice if the code that places it on the queue runs within a transaction that hits a database conflict and gets retried. Using an after-commit hook makes this pretty easy to deal with. I ended up creating a @salesforce_task decorator which encapsulates the logic for deferring the queueing to the right time as well as creating a Salesforce connection. See the code.

Then there's the other direction: polling Salesforce periodically for data that have been updated. I used to handle this with a cron job and "bin/instance run" script, but celery has support for periodic tasks so I figured I'd give that a try. The trick here is that this task needs to modify objects in the ZODB, so needs a full Zope environment set up when it runs in the celery worker process. Doing this is non-obvious but not especially difficult; I created another decorator called @zope_task which handles the dirty details. See the code.

Using Plone ids as a external id in Salesforce

To sync multiple changes to the same object between the 2 systems, we either need to store a Salesforce Id on the object in Plone, or a Plone id on the object in Salesforce. In the past I've stored a Salesforce Id in Plone, but this is annoying, because it means when you create a new object you can't push it to Salesforce asynchronously, because you need to wait for the id of the inserted object and store it. It turns out the Salesforce API has pretty good support for referring to objects by an "external id field" if you create one. So now we are adding an external id field called Plone_Id__c to our objects in Salesforce, and then it's pretty darn easy to push changes to Salesforce:

@salesforce_task
def do_upsert(sf, sobject_type, record_id, data):
    sobject = getattr(sf, sobject_type)
    return sobject.upsert(record_id, data)

@grok.subscribe(IContact, IObjectAddedEvent)
@grok.subscribe(IContact, IObjectModifiedEvent)
def handle_contact_modified(contact, event):
    data = {
        'FirstName': contact.first_name,
        'LastName': contact.last_name,
    }
    do_upsert.delay('Contact', 'Plone_Id__c/' + contact.UID(), data)

(In practice I have some extra checks to make sure that we don't call out to Salesforce while tests are running, or when those events are triggered due to pulling data in from Salesforce.)

You can even use an external id when referencing related objects, so we could update a committee that references the Contact who is chairing it like this:

@grok.subscribe(ICommittee, IObjectAddedEvent)
@grok.subscribe(ICommittee, IObjectModifiedEvent)
def handle_committee_modified(committee, event):
    data = {
        'Name': committee.title,
        'Chair__r': {  # this is the relation to the chair in Salesforce
            'Plone_Id__c': committee.chair  # in Plone we store a UID of the chair
        },
    }
    do_upsert.delay('Committee__c', 'Plone_Id__c/' + committee.UID(), data)

Next steps

This is all working very nicely and I'm *much* happier than I was with the old approaches. I'm also quite happy that the new approaches are for the most part Zope-agnostic, and could easily be modified to handle Salesforce integration from other Python-based platforms. Potential directions for future development (if someone wants to pay me to work on them) are:

  • Package up the decorators for creating celery tasks that run within Zope or do callouts to external webservices into a reusable library.
  • Create a higher-level, reusable Plone add-on with a UI for configuring synchronization between Plone content types and Salesforce objects.
2 comments

Goodbye, Groundwire. Hello, world!

by David Glick posted Sep 11, 2012 11:16 AM

After 5 years, I'm moving on from Groundwire to work independently.

In the summer of 2007 I graduated from college, moved to Seattle, and started working as a web developer for Groundwire, then called ONE/Northwest. This week, five years later, I'm both sad and excited to announce that I'm moving on.

Working at Groundwire has been a fantastic experience for which I am deeply grateful. I got to work with the best colleagues in the world, as well as some awesome clients whose missions I believe in. I learned a lot about shepherding nonprofit organizations through the process of doing a significant technology project. I was privileged to spend time pushing the Plone content management system to the next level and building deep integration between it and the Salesforce.com CRM platform. Groundwire is still doing great work to help nonprofits engage their constituents, and if you're currently looking for a great opportunity to combine your tech skills with a desire to work for good, I can wholeheartedly recommend applying for one of Groundwire's open positions.

But nevertheless, it's time for me to "level up" to the next challenge. After Thursday, I will start working as a freelance contractor specializing in Plone, Python, and web app development. I'm looking forward to working with and contributing my skills to some other great consultancies and clients, as well as learning new tools and techniques for constructing web-based solutions. And I'm looking forward to having a more flexible schedule with more chances to work on my own projects in addition to paid work.

The Plone content management system has grown dear to me during my time at Groundwire, and I will not be abandoning it. During the next few weeks leading up to the annual Plone conference, I plan to spend as much time as possible working on various Plone-related improvements that I have had scarce time for lately, including:

  • Attending the Sea Sprint and pushing forward the Deco layout system
  • Continuing to improve Dexterity's capabilities for building content types through the web
  • Helping prepare the release of Plone 4.3 (including completing the demise of KSS)

I'll be doing this work on a volunteer basis, but if you've benefited from my contributions to Plone and/or want to help make sure I continue to have time available to work on it, please consider leaving me a tip:

Following the Plone conference, I am scheduling projects for mid-October and beyond, and am especially interested in:

  • Chances to work with Plone, Pyramid, and/or other Python-based web frameworks—or train other developers on how to use them well
  • Web projects that are interactive rather than merely presenting information
  • Projects that expand what is possible for non-developers to accomplish using Plone
  • Projects that are driven in part by a "social good" mission rather than merely profit

If you have something like that I can assist with, please be in touch!

12 comments

Report from Cioppino Sprint 2012

by David Glick posted Mar 26, 2012 11:45 PM

Twelve Plonistas gathered together in Bodega Bay to write documentation and be merry.

Report from Cioppino Sprint 2012

Mr. Crabby

I've just returned from yet another memorable Plone event, the 2nd annual Cioppino Sprint. For the past 4 days, twelve of us gathered at a house in lovely Bodega Bay, CA for a weekend of fun, relaxation, and giving back to the Plone community.

The theme of the sprint was improving Plone's documentation and community infrastructure. On the first evening we gathered to brainstorm tasks, which were recorded on sticky notes to be placed on a scrum board indicating which tasks were pending, in progress, and complete. On Saturday, the sprinting began in earnest.

 

Sprint house

Tasks completed

Ross Patterson wrote a tutorial introducing the basics of zc.buildout. He also started investigating what it would take to let Plone be installed by Microsoft's free Web Platform Installer for Windows/IIS.

Luke Brannon and Ross wrote some CSS for reStructuredText "admonishments" (i.e. note/warning/etc. boxes) that will be added to plone.org for when reST documents are rendered. Luke also refactored and tested Windows installation instructions in the knowledgebase, reviewed KB articles for accurate version metadata, and started looking into sorting KB topic pages by Plone version in addition to modified date.

Spanky Kapanka learned how to contribute to the collective developer manual, then worked on writing several quick start guides to walk Plone beginners through specific tasks. He completed one on subclassing a content type and started one on creating a custom form.

Bill Deegan edited the introduction to the collective developer manual for clarity and to present Plone's capabilities in a more positive light alongside the fair warnings of complexity. He also renamed the files in the collective docs to have a .rst extension so they render when viewed on github, and tested Spanky's quick start.

Mike Cullerton learned git and reStructuredText, then worked on a "quick start" guide on customizing views and viewlets.

Liz Leddy worked on a new set of documentation of how to do Plone core development, which will live in the coredev buildout. She also did a lot of cleanup in Plone's trac and moved information to buildout.coredev and to a new, streamlined "Get Involved" landing page which replaces the front page of trac as an entry point to participating in the Plone community.

Steve McMahon worked on writing a Plone sysadmin manual discussing how to install and configure Plone for deployment scenarios on various platforms. I think he is about two-thirds done and has not published the document yet.

Tyler Randles (of ploneconf panda head fame) and I worked on improving the template for the knowledgebase documentation. We added a box at the top which shows a color-coded warning about the document's version compatibility, and a box at the bottom with a brief invitation to contribute to editing the knowledgebase (you too can help review KB articles to make sure their metadata is correct!). We also created mr.crabby, your blue crabby friend for Plone—can you find him on plone.org? (Hint: search plone.org for info on the Cioppino Sprint.) Tyler also worked on illustrations for Steve's manual.

I, David Glick, also added a version selector to the search forms in the plone.org documentation section, fixed a bug in the knowledgebase permissions so that articles are actually editable by the right people (i.e. anyone logged in), reviewed some of Liz's core dev docs, added i18n support to plone.supermodel and plone.registry's GenericSetup import handler in order to fix a Plone 4.2 blocker, and did some planning about how to better organize Dexterity's documentation.

Fulvio Casali reviewed and tried out the existing Dexterity documentation and created a detailed list of suggested improvements which will be very helpful.

Florian Friesdorf wrote instructions for setting up a Plone development environment using nix, and added support for paged searches to pas.plugins.ldap.

Eric Steele worked on getting Plone 4.2 ready for final release, including a lot of work on the new-style collections. (It needed some work on the date criteria for feature parity with the existing collections, and some work to make sure that Plone's default news and event collections are the new style.) By the end of the sprint Eric announced that there are no more blockers for the second release candidate!

Hanno Schlichting joined the #sprint IRC channel remotely and continued his quest for an improved ZCatalog. He suggested some catalog optimizations for plone.org which I will hopefully review soon, and implemented two long-desired new features: "not" queries and sorting results by multiple indexes.

LolaLola and Zoey, the dogs, sprinted along the beach and made themselves generally lovable.

Highlights

A highlight of the sprint for many of us was the food. Mike made homemade pizzas and introduced us to whoopie pies with habanero. Spanky prepared breakfast after wonderful breakfast, including his recently perfected Hollandaise for eggs benedict with crab cakes. For snacks we had Luke's guacamole and Spanky and Tyler's invention of tater tot-bacon-pepperjack cheese bites. I baked sourdough bread to go with Steve's cioppino featuring fresh crab, mussels and clams, and Liz made a great chili that was dubbed "Holy Crap!" Some of these recipes have been posted to the sprint mailing list so look for them there.

Food!

Another highlight for me was participating in my first "code dojo." This is a technique for sharing knowledge where a group gathers to complete a task surrounding a single big TV or projector screen. One person gets to control the keyboard, another person tells the typist what to do, and everyone else can be consulted for advice. We used this technique to work toward solving issue 12796, as an exercise in how to fix a bug in Plone core for those who hadn't done so before (and as a "functional test" of Liz's new documentation of this process). We didn't have time to complete the task, but I think everyone had fun and learned something new (within the first five minutes I learned a new bash keyboard shortcut!).

Beach exploring


One thing that did not happen at the sprint was a clear designation of a revitalized documentation team to make sure that our documentation is well-managed on an ongoing basis. Personally I feel that this is a role that is lacking in the community—Mikko and others are doing a fantastic job of getting people to add and update documentation in the collective developer manual, but I there is a need for a more focused manual and introductory documentation for people who are trying to learn Plone development for the first time rather than looking up particular tasks or topics—communally edited documentation is inevitably of varying quality and relevance, and newbies have no way to judge that. There is also a need to make sure that old documents are updated or marked as obsolete as appropriate. It's not entirely surprising that we've lacked this editing, as it is a thankless task (everyone can get excited about new documentation; someone always hates you when you delete something). I don't have answers but I hope we can brainstorm as a community about how to solve this problem (or maybe you can convince me I've diagnosed the wrong problem.)

But despite a lack of resolution of those questions, the sprint was a success in the realms of both fun and productivity. Thanks to the participants for a fantastic experience, to the Plone Foundation for sponsoring our lodging, and I look forward to seeing you at Cioppino 2013 if not before!

p.s. A special thanks to Spanky who provided a limo pickup and lodging to Fulvio and me after our train was canceled and we had to change travel plans at the last minute!

More photos

1 comment
David Glick

David Glick

I am a problem solver trying to make websites easier to build.

Currently I do this in my spare time as a member of the Plone core team, and during the day as an independent web developer specializing in Plone and custom Python web applications.