Building a version of NumPy that links with a specific Visual Studio Runtime Library

In the previous blog article we showed how to build a Mingw-w64 toolchain that links with a specific runtime. In this article we will use this toolchain, along with a locally built Python installation, to generate a version of NumPy that links with msvcrt110.dll and can subsequently be used in Maya 2016.

Building Python 2.7.11 with Visual Studio 2012

This will allow building most Visual Studio compatible plugins, but will also help while building with the Mingw-w64 toolchain. You need to install a few packages to deal with the build scripts:

• Subversion
• Mercurial
• Perl

 Then download the latest Python source tarball and extract it in your work area.

Once Python is extracted:

• open a command prompt and run Python-2.7.11\PCBuild\get_externals.bat
• open Python-2.7.11\PCBuild\pcbuild.sln in Visual Studio 2012 and let it upgrade all projects
• select the Release x64 configuration and press F7 to build the whole solution

The last step might fail to build the Tk/TkInter modules, but we will not need them, and the binairies compiled in Python-2.7.11\PCBuild\amd64\ will be sufficient. We need to add a few touches before it can be used to build packages. All the commands below are done in the mingw-w64 we prepared in the previous blog entry. It can be opened by executing C:\msys64\mingw64_shell.bat. The downloaded files mentionned below were previously fetched in the MSys home folder, C:\msys64\home\jgamache in this case.

1- Apply the mingw64 definition patch from Python issue 11723 to the newly built Python:

$ cd /c/Workspace/Python-2.7.11
$ patch -p2 < ~/Downloads/mingw64.patch

2- Apply the MS_WIN64 declaration patch from Python issue 4709:

$ patch -p0 < ~/Downloads/mingw-w64.patch

3- At the top of /c/Workspace/Python-2.7.11/Lib/distutils/cygwinccompiler.py add the lines in red to the get_msvcr( ) function:

        elif msc_ver == '1500':
            # VS2008 / MSVC 9.0
            return ['msvcr90']
        elif msc_ver == '1600':
            # VS2010 / MSVC 10.0
            return ['msvcr100']
        elif msc_ver == '1700':
            # VS2012 / MSVC 11.0
            return ['msvcr110']
        else:
            raise ValueError("Unknown MS Compiler version %s " % msc_ver)

4- Generate the python27.lib file to allow linking Python modules with the Mingw-w64 toolchain:

$ cd /c/Workspace/Python-2.7.11/PCbuild/amd64/
$ gendef python27.dll
$ dlltool -D python27.dll -d python27.def -l libpython27.a
$ mkdir /c/Workspace/Python-2.7.11/libs
$ cp *.lib /c/Workspace/Python-2.7.11/libs
$ cp python27.def libpython27.a /c/Workspace/Python-2.7.11/libs

5- Install setuptools:

$ cd ~/Downloads
$ wget https://bootstrap.pypa.io/ez_setup.py
$ python ez_setup.py

Remove any mention of -mno-cygwin in build scripts since the option is obsolete:

$ perl -pi -e 's/-mno-cygwin//g' /c/Workspace/Python-2.7.11/Lib/distutils/cygwinccompiler.py

Building NumPy

Now that we have a proper toolchain and Python interpreter, we are ready to build NumPy. Open a Mingw-w64 shell from C:\msys64\mingw64_shell.bat and set the PATH environment variable to use only the Mingw-w64 binaries and the new Python:

$ export PATH=/mingw64/bin:/usr/local/bin:/usr/bin:/bin:/c/Workspace/Python-2.7.11/PCbuild/amd64

Extract and build OpenBLAS with support for multiple processors:

$ mkdir ~/Workspace
$ cd ~/Workspace
$ git clone https://github.com/xianyi/OpenBLAS.git
$ cd OpenBLAS
$ make DYNAMIC_ARCH=1 -j40

Get the NumPy 1.9.2 tarball from SourceForge and extract it to your Mingw-w64 workspace, then apply Carl Kleffner's excellent Mingw-w64 patch:

$ cd ~/Workspace
$ tar -xzf ~/Downloads/numpy-1.9.2.tar.gz
$ patch -p0 < ~/Downloads/numpy-1.9.2.patch
$ cd numpy-1.9.2/
$ vim site.cfg

Add the following lines to the site.cfg file you just opened in VIM:

[openblas]
libraries = openblas
library_dirs = C:/msys64/home/jgamache/Workspace/OpenBLAS
include_dirs = C:/msys64/home/jgamache/Workspace/OpenBLAS

Add this code at the beginning of numpy-1.9.2/numpy/__init__.py just after the "from __future__ import ..." line:

# Numpy-Mingw-w64: prepend the path of the Mingw DLLs to os.environ['PATH']
def _add2path():
	import os
	if os.name != 'nt':
		return
	try:
		path = os.path.join(os.path.abspath(os.path.dirname(__file__)), 'core')
		if path not in os.environ.get('PATH', ''):
			os.environ['PATH'] = os.pathsep.join((path, os.environ.get('PATH', '')))
	except Exception:
		pass

_add2path()
del _add2path

You can now build and package NumPy:

$ python setup.py build
$ python setup.py bdist
$ cd dist/
$ unzip numpy-1.9.2.win-amd64.zip  # Use pacman -S unzip if not found
$ cp /mingw64/bin/libgfortran-3.dll Workspace/Python-2.7.11/Lib/site-packages/numpy/core/
$ cp /mingw64/bin/libquadmath-0.dll Workspace/Python-2.7.11/Lib/site-packages/numpy/core/
$ cp /mingw64/bin/libgcc_s_seh-1.dll Workspace/Python-2.7.11/Lib/site-packages/numpy/core/
$ cp /mingw64/bin/libwinpthread-1.dll Workspace/Python-2.7.11/Lib/site-packages/numpy/core/
$ cp ~/Workspace/OpenBLAS/libopenblas.dll Workspace/Python-2.7.11/Lib/site-packages/numpy/core/
# Fix the Path. Using Python since we target Maya2016
$ mkdir Python
$ mv Workspace/Python-2.7.11/* Python
$ zip -r9 numpy-1.9.2.win-amd64-Maya2016.zip Python/

Let's test:

$ cd ~
$ wget https://raw.github.com/pypa/pip/master/contrib/get-pip.py
$ python get-pip.py
$ python -m pip install -U pip
$ python -m pip install nose
$ python -i
Python 2.7.11 (default, Dec 17 2015, 11:38:06) [MSC v.1700 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path.append("C:\\msys64\\home\\jgamache\\Workspace\\numpy-1.9.2\\dist\\Python\\Lib\\site-packages")
>>> import numpy
>>> numpy.test()
Running unit tests for numpy
NumPy version 1.9.2
NumPy is installed in C:\msys64\home\jgamache\Workspace\numpy-1.9.2\dist\Python\Lib\site-packages\numpy
Python version 2.7.11 (default, Dec 17 2015, 11:38:06) [MSC v.1700 64 bit (AMD64)]
nose version 1.3.7
...

----------------------------------------------------------------------
Ran 5199 tests in 21.908s

OK (KNOWNFAIL=10, SKIP=20)

You now have a NumPy with a powerful BLAS library ready to be used in Maya 2016. Enjoy!