Key computing tools in Python for scientific research

By the late 1990s, discussions appeared on Matrix-SIG expressing a desire for a complete scientific data analysis environment¹⁶. Travis Oliphant, a PhD student at the Mayo Clinic, released a number of packages^17,18 that built on top of the Numeric array package, and provided algorithms for signal processing, special functions, sparse matrices, quadrature, optimization, fast Fourier transforms and more. One of these packages, Multipack (http://pylab.sourceforge.net/multipack.html↗), was a set of extension modules that wrapped Fortran and C libraries to solve nonlinear equations and least-squares problems, integrate differential equations and fit splines. Robert Kern, then an undergraduate student (and currently a SciPy core developer), provided compilation instructions under Windows. Around the same time, Pearu Peterson, a PhD student from Estonia, released F2PY¹⁹, a command line tool for binding Python and Fortran codes, and wrote modules for linear algebra and interpolation. Eric Jones, while a graduate student at Duke University, wrote packages to support his dissertation, including a parallel job scheduler and genetic optimizer. Gary Strangman, a postdoctoral fellow at Harvard Medical School, published several descriptive and inferential statistical routines²⁰.

With a rich programming environment and a numerical array object in place, the time was ripe for the development of a full scientific software stack. In 2001, Eric Jones and Travis Vaught founded Enthought Scientific Computing Solutions (now Enthought, Inc.) in Austin, Texas, USA. To simplify the tool stack, they created the SciPy project, centered around the SciPy library, which would subsume all the above-mentioned packages. The new project quickly gained momentum, with a website and code repository²¹ appearing in February, and a mailing list announced²² in June 2001. By August 2001, a first release was announced²³, an excerpt of which is shown in Box 1. In September, the first documentation was published²⁴. The first SciPy workshop²⁵ was held in September 2002 at Caltech—a single track, two-day event with 50 participants, many of them developers of SciPy and surrounding libraries.

At this point, scientific Python started attracting more serious attention; code that started as side projects by graduate students had grown into essential infrastructure at national laboratories and research institutes. For example, Paul Dubois at Lawrence Livermore National Laboratory (LLNL) took over the maintenance of Numeric and funded the writing of its manual²⁶, and the Space Telescope Science Institute (STScI), which was in charge of Hubble Space Telescope science operations, decided to replace their custom scripting language and analysis pipeline with Python²⁷. As STScI continued to use Python for an increasingly large portion of the Hubble Space Telescope data analysis pipeline, they encountered problems with the Python numerical array container. Numeric, the original array package, was suitable for small arrays, but not for the large images processed by STScI. With the Numeric maintainer’s blessing, the decision was made to write NumArray²⁸, a library that could handle data on a larger scale. Unfortunately, NumArray proved inefficient for small arrays, presenting the community with a rather unfortunate choice. In 2005, Travis Oliphant combined the best elements of Numeric and NumArray, thereby solving the dilemma. NumPy 1.0 was released²⁹ in October 2006, paving the way for the reunified scientific Python community to mature.

By the middle to late 2000s, SciPy was starting to mature after a long phase of significant growth and adoption. The scope of the SciPy library narrowed, while the breadth of the ecosystem grew through a new type of auxiliary package: the scikit (https://www.scipy.org/scikits.html↗), a complementary library developed outside SciPy, allowing for more rapid exploration of experimental ideas while maintaining familiar style and development methodology. In SciPy itself, tooling, development, documentation and release processes became more professional. The library was expanded carefully, with the patience affordable in open-source projects and via best practices, which are increasingly common in the scientific Python ecosystem and industry³⁰.

Early versions of SciPy had minimal documentation, but this began to change with the 2006 release of a Guide to NumPy¹. In 2007, Sphinx³¹ made it possible to render hypertext and PDF documents automatically from plain text (docstrings) interspersed with Python code, and in 2008, pydocweb³² enabled collaborative documentation development in a wiki-like fashion. The SciPy Documentation Project^33,34 used these tools to complete documentation of SciPy’s user-facing functionality: offering t-shirts to contributors from around the world in exchange for high-quality text, it collected contributions from over 75 people to produce an 884-page manual³⁵. Since then, SciPy has remained committed to maintaining high-quality documentation as part of the normal development cycle.

In the early SciPy workshops, recurrent topics reflected the state of development, with emphasis being placed on the underlying array package, plotting, parallel processing, acceleration/wrapping and user interfaces. By 2004, presentations about the application of SciPy to scientific problems began to appear. The event also started to draw in more keynote speakers from outside the community, such as Guido van Rossum (creator of Python, 2006), Ivan Krstić (One Laptop per Child, 2007), Alex Martelli (Google, 2008) and Peter Norvig (Google Research, 2009). The informal workshop grew from a small gathering of core developers into an international conference with hundreds of attendees, increased funding, a published proceedings and scholarships for attending students. By 2010, the US SciPy conference had multiple tracks, and satellite conferences were being organized by volunteers elsewhere, such as EuroSciPy (since 2008) and SciPy India (since 2009). Special sessions and minisymposia dedicated to scientific Python began appearing at many other events. For example, a three-part minisymposium organized for International Conferences on Computational Science and Engineering (CSE) 2009 was featured in SIAM News³⁶.

In 2007, Python had a strong enough presence in science and engineering that the editors of IEEE Computing in Science and Engineering solicited a special issue about Python in science³⁷, edited by Paul Dubois. However, Python was still sufficiently niche that the average reader would need additional information to decide whether it would be useful in their own work. The follow-up March/April 2011 Python for Scientists and Engineers special issue³⁸ focused more on the core parts of the scientific Python ecosystem³⁹ including NumPy², Cython⁴⁰ and Mayavi⁴¹. Python became so pervasive that journals began publishing domain-specific special issues. For example, in 2015, Frontiers in Neuroinformatics published a collection of 25 articles—covering topics including modeling and simulation, data collection, electrophysiology, visualization as well as stimulus generation and presentation—called Python in Neuroscience⁴².

As of February 2019, the SciPy library consists of nearly 600,000 lines of open-source code organized in 16 subpackages summarized in Box 2. The development team and community currently interact and operate primarily on GitHub, an online version control and task management platform. Over 110,000 GitHub repositories and 6,500 packages depend on SciPy⁴³. Some of the major feature highlights from the three years preceding SciPy 1.0 are discussed in the “Key technical improvements” section below, and milestones in its history are highlighted in Fig. 1.

SciPy provides fundamental algorithms for scientific computing. The breadth of its scope was derived from the guide to available mathematical software (GAMS) classification system⁴⁴. In areas that move relatively slowly, for example, linear algebra, SciPy aims to provide complete coverage. In other areas it aims to provide fundamental building blocks while interacting well with other packages specialized in that area. For example, SciPy provides what one expects to find in a statistics textbook (probability distributions, hypothesis tests, frequency statistics, correlation functions, and more), whereas Statsmodels⁴⁵ provides more advanced statistical estimators and inference methods, scikit-learn³ covers machine learning, and PyMC3⁴⁶, emcee⁴⁷ and PyStan (http://mc-stan.org↗) cover Bayesian statistics and probabilistic modeling. scikit-image⁴ provides image processing capabilities beyond SciPy’s ndimage, SymPy⁴⁸ provides a Python interface for symbolic computation, and sparse.csgraph and spatial offer basic tools for working with graphs and networks compared to specialized libraries such as NetworkX⁴⁹.

We use the following criteria to determine whether to include new functionality in SciPy:

In terms of software systems and architecture, SciPy’s scope matches NumPy’s: algorithms for in-memory computing on single machines, with support for a wide range of data types and process architectures. Distributed computing and support for graphics processing units (GPUs) were explicitly out of scope at the 1.0 release point, but this has been revised in our roadmap (see Discussion).

According to analysis using the linguist library (https://github.com/github/linguist↗), SciPy is approximately 50% Python, 25% Fortran, 20% C, 3% Cython and 2% C++, with a dash of TeX, Matlab, shell script and Make. The distribution of secondary programming languages in SciPy is a compromise between a powerful, performance-enhancing language that interacts well with Python (that is, Cython) and the usage of languages (and their libraries) that have proven reliable and performant over many decades.

Fortran, despite its age, is still a high-performance scientific programming language with continued contemporary usage⁵⁰. Thus, we wrap the following excellent, field-tested Fortran libraries to provide Python convenience while benefiting from their performance: QUADPACK⁵¹ and ODEPACK⁵² for numerical integration and solution of initial value problems; FITPACK⁵³, ODRPACK⁵⁴ and MINPACK⁵⁵ for curve-fitting and least-squares minimization; FFTPACK^56,57 for performing Fourier transforms; ARPACK⁵⁸ for solving eigenvalue problems; ALGORITHM 644 (ref. ⁵⁹) for computing Bessel functions; and CDFLIB⁶⁰ for evaluating cumulative density functions.

Rounding out the top three languages in SciPy is C, which is also extremely well-established over several decades⁶¹ of scientific computing. The C libraries that we wrap in SciPy include trlib⁶² for optimization, SuperLU^63,64 for solving sparse linear systems, Qhull⁶⁵ for computational geometry and Cephes (http://www.netlib.org/cephes/↗) for special functions.

Cython has been described as a creole language that mixes the best parts of Python and lower-level C/C++ paradigms⁴⁰. We often use Cython as a glue between well-established, low-level scientific computing libraries written in C/C++ and the Python interface offered by SciPy. We also use Cython to enable performance enhancements in Python code, especially for cases where heavily used inner loops benefit from a compiled code with static typing.

For implementing new functionality, Python is the still the language of choice. If Python performance is an issue, then we prefer the use of Cython followed by C, C++ or Fortran (in that order). The main motivation for this is maintainability: Cython has the highest abstraction level, and most Python developers will understand it. C is also widely known, and easier for the current core development team to manage than C++ and especially Fortran.

The position that SciPy occupies near the foundation of the scientific Python ecosystem is such that adoption of new languages or major dependencies is generally unlikely; our choices are strongly driven by long-term stability. GPU acceleration, new transpiling libraries and the latest JIT compilation approaches (for example, Numba⁶⁶) are very powerful but have traditionally fallen outside the remit of the main SciPy library. That said, we have recently increased our efforts to support compatibility with some of these options, and our full test suite passed with the PyPy JIT compiler⁶⁷ at the 1.0 release point.

The API for SciPy consists of approximately 1,500 functions and classes. Our policy for evolving the API over time is that new functionality can be added, while removing or changing existing functionality can only be done if the benefits exceed the (often significant) costs to users and only after giving clear deprecation warnings to those users for at least one year. In general, we encourage changes that improve clarity in the API of the library but strongly discourage breaking backward compatibility, given our position near the base of the scientific Python computing stack.

In addition to the Python API, SciPy has C and Cython interfaces. Therefore, we also have to consider the application binary interface (ABI). This ABI has been stable for a long time, and we aim to evolve it only in a backward-compatible way.

SciPy has provided special functions and leveraged basic linear algebra subprograms (BLAS) and linear algebra package (LAPACK)⁷⁶ routines for many years. SciPy now additionally includes Cython⁴⁰ wrappers for many BLAS and LAPACK routines (added in 2015) and the special functions provided in the scipy.special subpackage (added in 2016), which are available in scipy.linalg.cython_blas, scipy.linalg.cython_lapack and scipy.special.cython_special, respectively. When writing algorithms in Cython, it is typically more efficient to call directly into the libraries SciPy wraps rather than indirectly, using SciPy’s Python APIs. These low-level interfaces for Cython can also be used outside of the SciPy codebase to gain access to the functions in the wrapped libraries while avoiding the overhead of Python function calls. This can give performance gains of one or two orders of magnitude for many use cases.

Developers can also use the low-level Cython interfaces without linking against the wrapped libraries⁷⁷. This lets other extensions avoid the complexity of finding and using the correct libraries. Avoiding this complexity is especially important when wrapping libraries written in Fortran. Not only can these low-level wrappers be used without a Fortran compiler, they can also be used without having to handle all the different Fortran compiler ABIs and name mangling schemes.

Most of these low-level Cython wrappers are generated automatically to help with both correctness and ease of maintenance. The wrappers for BLAS and LAPACK are primarily generated using type information that is parsed from the BLAS and LAPACK source files using F2PY¹⁹, though a small number of routines use hand-written type signatures instead. The input and output types of each routine are saved in a data file that is read at build time and used to generate the corresponding Cython wrapper files. The wrappers in scipy.special.cython_special are also generated from a data file containing type information for the wrapped routines.

Since SciPy can be built with LAPACK 3.4.0 or later, Cython wrappers are only provided for the routines that maintain a consistent interface across all supported LAPACK versions. The standard BLAS interface provided by the various existing BLAS libraries is not currently changing, so changes are not generally needed in the wrappers provided by SciPy. Changes to the Cython wrappers for the functions in scipy.special follow corresponding changes to the interface of that subpackage.

The scipy.optimize subpackage provides functions for the numerical solution of several classes of root finding and optimization problems. Here we highlight recent additions through SciPy 1.0.

The scipy.stats subpackage contains more than 100 probability distributions: 96 continuous and 13 discrete univariate distributions, and 10 multivariate distributions. The implementation relies on a consistent framework that provides methods to sample random variates, to evaluate the cumulative distribution function (CDF) and the probability density function (PDF), and to fit parameters for every distribution. Generally, the methods rely on specific implementations for each distribution, such as a closed-form expression of the CDF or a sampling algorithm, if available. Otherwise, default methods are used based on generic code, for example, numerical integration of the PDF to obtain the CDF. Key recent distributions added to scipy.stats include the histogram-based distribution in scipy.stats.rv_histogram and the multinomial distribution in scipy.stats.multinomial (used, for example, in natural language processing⁸³).

Historically, SciPy relied heavily on the venerable FITPACK Fortran library by P. Dierckx^53,84 for univariate interpolation and approximation of data, but the original monolithic design and API for interaction between SciPy and FITPACK was limiting for both users and developers.

Implementing a new, modular design of polynomial interpolators was spread over several releases. The goals of this effort were to have a set of basic objects representing piecewise polynomials, to implement a collection of algorithms for constructing various interpolators, and to provide users with building blocks for constructing additional interpolators.

At the lowest level of the new design are classes that represent univariate piecewise polynomials: PPoly (SciPy 0.13)⁸⁵, BPoly (SciPy 0.13) and BSpline (SciPy 0.19)⁸⁶, which allow efficient vectorized evaluations, differentiation, integration and root-finding. PPoly represents piecewise polynomials in the power basis in terms of breakpoints and coefficients at each interval. BPoly is similar and represents piecewise polynomials in the Bernstein basis (which is suitable, for example, for constructing Bézier curves). BSpline represents spline curves, that is, linear combinations of B-spline basis elements⁸⁷.

In the next layer, these polynomial classes are used to construct several common ways of interpolating data: CubicSpline (SciPy 0.18)⁸⁸ constructs a twice differentiable piecewise cubic function, Akima1DInterpolator and PCHIPInterpolator implement two classic prescriptions for constructing a C¹ continuous monotone shape-preserving interpolator^89,90.

SciPy adopted an official governance document (https://docs.scipy.org/doc/scipy/reference/dev/governance/governance.html↗) on August 3, 2017. A steering council, currently composed of 18 members, oversees daily development of the project by contributing code and reviewing contributions from the community. Council members have commit rights to the project repository, but they are expected to merge changes only when there are no substantive community objections. The chair of the steering council, Ralf Gommers, is responsible for initiating biannual technical reviews of project direction and summarizing any private council activities to the broader community. The project’s benevolent dictator for life, Pauli Virtanen, has overruling authority on any matter, but is expected to act in good faith and only exercise this authority when the steering council cannot reach agreement.

SciPy’s official code of conduct was approved on October 24, 2017. In summary, there are five specific guidelines: be open to everyone participating in our community; be empathetic and patient in resolving conflicts; be collaborative, as we depend on each other to build the library; be inquisitive, as early identification of issues can prevent serious consequences; and be careful with wording. The code of conduct specifies how breaches can be reported to a code of conduct committee and outlines procedures for the committee’s response. Our diversity statement “welcomes and encourages participation by everyone.”

The SciPy project has ~100 unique contributors for every 6-month release cycle. Anyone with the interest and skills can become a contributor; the SciPy contributor guide (https://scipy.github.io/devdocs/dev/contributor/contributor_toc.html↗) provides guidance on how to do that. In addition, the project currently has 15 active (volunteer) maintainers: people who review the contributions of others and do everything else needed to ensure that the software and the project move forward. Maintainers are critical to the health of the project⁹³; their skills and efforts largely determine how fast the project progresses, and they enable input from the much larger group of contributors. Anyone can become a maintainer, too, as they are selected on a rolling basis from contributors with a substantial history of high-quality contributions.

The development cost of SciPy is estimated in excess of 10 million dollars by Open Hub (https://www.openhub.net/p/scipy/estimated_cost↗). Yet the project is largely unfunded, having been developed predominantly by graduate students, faculty and members of industry in their free time. Small amounts of funding have been applied with success: some meetings were sponsored by universities and industry, Google’s Summer of Code program supported infrastructure and algorithm work, and teaching grant funds were used early on to develop documentation. However, funding from national agencies, foundations and industry has not been commensurate with the enormous stack of important software that relies on SciPy. More diverse spending to support planning, development, management and infrastructure would help SciPy remain a healthy underpinning of international scientific and industrial endeavors.

The scientific Python ecosystem includes many examples of domain-specific software libraries building on top of SciPy features and then returning to the base SciPy library to suggest and even implement improvements. For example, there are common contributors to the SciPy and Astropy core libraries⁹⁴, and what works well for one of the codebases, infrastructures or communities is often transferred in some form to the other. At the codebase level, the binned_statistic functionality is one such cross-project contribution: it was initially developed in an Astropy-affiliated package and then placed in SciPy afterward. In this perspective, SciPy serves as a catalyst for cross-fertilization throughout the Python scientific computing community.

Further information on research design is available in thelinked to this article. Nature Research Reporting Summary

Raw data for Fig. 2 are available at https://github.com/tylerjereddy/scipy-cov-track↗, and raw data for Fig. 3 are available at https://github.com/scipy/scipy-articles/tree/master/scipy-1.0/supporting_info/asv_bench/cKDTree↗.

All SciPy library source code is available in the SciPy GitHub repository, https://github.com/scipy/scipy↗.