Cosmic Serpent: DNA and the great Python 3 Migration

(with apologies to Jeremy Narby)

Like any good software engineering team, we’re always updating our dependencies to make sure we’re running the most secure, bug-free versions. Recently, we tackled our oldest dependency: Python 2. Python 3 was released nearly 10 years ago, in December 2008 — long before Color was even founded! — but has been slow to unseat Python 2 as the industry standard. As of 2017, an estimated half of all Python projects still hadn’t made the switch.

We’ve been thinking about Python 3 since the early days of Color, hoping to become relatively early adopters. Dependencies are always the first obstacle: back in 2014, common bioinformatics packages had no plan for Python 3 compatibility. We launched Color on Python 2, and over the years we periodically checked in on the compatibility of our dependencies, dreaming of Unicode strings, sane division, type annotations, and more. We’d inevitably get stuck when we learned that some small but important dependency still wasn’t Python 3 compatible. Plus, time is precious, so we deferred upgrading in favor of more pressing and immediate goals.

During our August 2017 Fixit Week, we took another pass through our dependencies and were pleasantly surprised to find that of the 130+ packages we relied on, all but a handful were Python 3 compatible. Fabric, Ansible, and Supervisor were important exceptions, but even those now had a path forward: we found a Python 3 fork of Fabric, and could install Ansible and Supervisor outside of our runtime virtualenv. So, with our dependencies in good shape, we began planning a full migration of our main repo.

According to cloc, we’d need to migrate more of 100,000 lines of non-test code, with roughly the same again in tests. We studied previous migration stories, advice, and popular tools for automating much of the work of updating, like 2to3 (links to recommended reading: [1, 2, 3]). We appreciated benefitting from wisdom shared by others, so we want to share our own story, along with a few additional tips and notes that we haven’t seen elsewhere.

Broadly speaking, there are two approaches to upgrading a Python 2 codebase to Python 3:

• Upgrade incrementally, maintaining Python 2 compatibility while gradually adding Python 3 support. Eventually you switch your interpreter, and your code continues to work as usual. This approach relies on libraries such as modernize, six, and future to bridge the gaps.
• Upgrade the entire codebase, all at once. This approach relies on running 2to3, which upgrades your code without attempting to maintain compatibility.

For the most part we chose the second option. Many organizations spend months (or years!) on painfully slow migrations, an ongoing tax we sought to avoid. With our next Fixit Week just around the corner, we decided it would be more expedient to start a separate branch for Python 3, using Fixit Week as an opportunity to institute a brief code freeze so we could safely merge the Python 3 branch into master and test it extensively before deploying to production.

A few weeks ahead of the planned switchover, we worked on getting all of our unit tests passing in the Python 3 branch, backporting as many changes as possible into master to minimize the difference between the branches. Our workflow was:

1. At the beginning of the week, checkout a new branch based off latest master.
2. Apply 2to3 fixers one at a time.
3. Rebase all changes from previous branch onto new branch.
4. Run tests and fix things.
5. Backport safe fixes into master.
6. Goto 1 and repeat.

Thanks to relatively thorough test coverage, we found that we were able to safely backport many changes to master (still running Python), either directly or via limited/judicious use of six:

• The types, tuple_params, reduce, numliterals, next, funcattrs, except, and asserts 2to3 fixers, which resulted in Python 2 compatible code in our codebase.
• Exception.message, which was already deprecated as of Python 2.7.
• Converting print keywords to function calls, by adding from __future__ import print_function and a precommit hook to enforce using the function.
• The import fixer, along with switching to absolute imports with from __future__ import absolute import.
• Upgrading libraries to Python 3 compatible versions.
• Comparators, i.e. the cmp keyword.
• Using iterators directly where possible, notably dict.iteritems instead of items. Much of this was just removing unnecessary list() calls added by the iteritems fixer.
• The division operator: we took a simple approach, grepping for /, filtering out false positives (lots of them), confirming that a float return value was acceptable or changing the operator to //, and finally adding from __future__ import division.

These backported changes ran in production, in advance of our switchover, long enough that we were able to iron out the few wrinkles we had initially overlooked. Smooth sailing so far.

As Fixit Week began, we deployed our release candidate Python 3 branch to our internal staging environment and began aggressive manual QA together with our colleagues. We hoped that our automated testing was sufficient to catch any non-trivial regressions, but of course, hope is not a strategy. We were pleased to discover zero regressions in the most sensitive and carefully validated parts of our codebase, especially the bioinformatics pipeline and our genetic report/result generator, both of which we backtest extensively against historic data.

We did uncover two minor regressions in rarely exercised non-clinical codepaths, but somewhat gratifyingly, these weren’t caused by the Python 3 migration: they had been introduced the previous week. That’s a successful milestone, in our book. (And of course, we fixed and updated test coverage accordingly!)

We were ready. With bated breath, we deployed our initial Python 3 release on Wednesday. Thanks to our test plan, things went quite smoothly, with no external-facing customer impact. Woo!

In the spirit of Color’s learning culture, we note that we did encounter a few character encoding issues with asynchronous background tasks that lacked sufficient test coverage, which were easy to diagnose and fix. The most significant issue, and a tip which we haven’t seen shared elsewhere, is that we didn’t test running Python 2 and Python 3 side by side (as happens naturally when multiple instances are being updated mid-deploy).

Specifically, we cache heavily in production, and Python 2 was unable to read some cached objects stored by Python 3 because they were serialized with a newer version of pickle. We also saw this in other places that our code communicated across processes, notably arguments to Celery tasks that were pickled by an application server and unpickled by a Celery worker.

The migration’s biggest operational cost was, perhaps surprisingly, troubleshooting a few of our colleagues’ local development environments as they made the switch. We engineers are often particular about how we like our environment configured, and as usual, XKCD hits a little too close to home here. Not a big deal at Color — we’re a small team and can afford to help our colleagues individually get up and running in the new environment — but larger engineering organizations should plan accordingly.

Overall, the migration was a sound investment for Color engineering, one which we hope will benefit our productivity and the onboarding of our future colleagues — the next generation of Color engineers don’t need to learn legacy Python quirks! Coding with Python 3 might not feel like living in the future, but we’re now at least squarely in the present.

Have you attempted a Python 3 migration? Leave a response and share your team’s experience — tips, tricks, or pitfalls to be avoided.

Thanks to TP Wong, Zach Langley, Jeremy Ginsberg, and everyone else in Color’s engineering and product teams who went above and beyond the call of duty to make this happen.

We’re hiring! If you’re a software engineer hoping to apply your skills in the service of a greater mission, while embracing modern software development practices and technologies, drop us a line! Check out our other recent engineering posts to get a sense of how our team operates, or visit color.com/careers for more information.