Upgrading from Python 2 to Python 3

I've always liked to think of myself as an early adopter. However to date, I've been using Python 2 and it's been bugging me. We're up to Python 3.5 already.

I think the biggest barrier for me has always been that OS X's default is still 2.7 but it's not like that's difficult to get around. I'm mean, I am a mad hacker after all!

Getting Python 3 on your OS X machine is super easy thanks to Homebrew:

§ brew install python3

Don't worry it just installs Python 3 right next to Python 2.7.

To find out where it is installed it's as simple as:

§ which python3

And once you know that you can easily setup a virtualenvwrapper that used Python 3 so you never have to think about it again...

§ mkvirtualenv --python=/usr/bin/python3 nameOfEnvironment

My next step came from a hot tip from the Neck Beard Republic's fantastic screen cast on Converting Python2 to Python3. He suggests you use a tool called 2to3 which scans a specified file or directory for changes required to upgrade.

So what work was required to upgrade my app from Python 2 to 3?

There were a couple of print statements that I needed to turn into functions. Obvs. I'm still working on the required muscle memory for that one.

Then there was relative imports. The 2to3 scan recommended adding a dot (.) to a range of imports where I'm importing modules from my app. I'm still getting my head around this one and when I actually ran my code and tests I found I had to include the module name as well as the dot.

So instead of:

from .base import BaseTestCase

I found I had to:

from tests.base import BaseTestCase

Strangely I haven't had to do that for the scrapers though. As I say, I'm not sure I've fully got my head around that one.

Then there was a change in the urllib package but that was a simple fix just by changing the imports from:

from urllib import quote

to:

from urllib.parse import quote

There were also a few unicodes but that was more or less a bulk find and replace of 'unicode' for 'str' as str has replaced unicode in Python 3 where unicode is now (quite rightly) the default encoding.

The zip() function now returns a generator rather than a list. I'm not sure if it would have made any difference but I changed

query = zip(conditions[0::2], conditions[1::2])

to

query = list(zip(conditions[0::2], conditions[1::2]))

And the last thing I need to change was the way I handle the response of a csv file from the Requests module hands a file off to the csv.DictReader() method. I was getting an error saying:

_csv.Error: iterator should return strings, not bytes (did you open the file in text mode?)

As it turns out this is related to the encoding changes in Python 3. So what, in Python 2, looked like this:

def getCsvData(self, file):
    ''' Returns dictionary of CSV Data '''
    csvfile = requests.get(file, stream=True)
    return csv.DictReader(csvfile.raw)

Now looks like this:

def getCsvData(self, file):
    ''' Returns dictionary of CSV Data '''
    csvfile = requests.get(file, stream=True)
    return csv.DictReader(StringIO(csvfile.text))

(with an import of StringIO from the io module).

The final thing left to do was to reinstall the relevant Python modules using pip. This primarily meant that wsgiref was excluded this time which I'm pretty sure was/is part of the Flask dependencies and presumably not required with Python 3.

Realistically, this took be about 45 mins and has, so far, been pretty hassle free. A couple of Google Searches to figure out the StringIO thing was the most onerous and now I'm back to being an early adopter. There's pretty handy guid here and you can see most of the changes I had to make in this commit.

Update:

I forgot to mention one other error that I got that it is probably worth mentioning.

When running my tests I got an error saying:

AssertionError: Popped wrong request context.

It turns out this is related to Flask-Test, context preservation and Python 3 (I couldn't find a clear reason why in anything I read). To fix it I just added the following to my manage.py file:

app.config['PRESERVE_CONTEXT_ON_EXCEPTION'] = False