PyPy (Posts by Richard Plangger)

Native profiling in VMProf

Richard Plangger — Sat, 01 Apr 2017 14:17:00 GMT

We are happy to announce a new release for the PyPI package vmprof.
It is now able to capture native stack frames on Linux and Mac OS X to show you bottle necks in compiled code (such as CFFI modules, Cython or C Python extensions). It supports PyPy, CPython versions 2.7, 3.4, 3.5 and 3.6. Special thanks to Jetbrains for funding the native profiling support.

What is vmprof?

If you have already worked with vmprof you can skip the next two section. If not, here is a short introduction:

The goal of vmprof package is to give you more insight into your program. It is a statistical profiler. Another prominent profiler you might already have worked with is cProfile. It is bundled with the Python standard library.

vmprof's distinct feature (from most other profilers) is that it does not significantly slow down your program execution. The employed strategy is statistical, rather than deterministic. Not every function call is intercepted, but it samples stack traces and memory usage at a configured sample rate (usually around 100hz). You can imagine that this creates a lot less contention than doing work before and after each function call.

As mentioned earlier cProfile gives you a complete profile, but it needs to intercept every function call (it is a deterministic profiler). Usually this means that you have to capture and record every function call, but this takes an significant amount time.

The overhead vmprof consumes is roughly 3-4% of your total program runtime or even less if you reduce the sampling frequency. Indeed it lets you sample and inspect much larger programs. If you failed to profile a large application with cProfile, please give vmprof a shot.

vmprof.com or PyCharm

There are two major alternatives to the command-line tools shipped with vmprof:

A web service on vmprof.com
PyCharm Professional Edition

While the command line tool is only good for quick inspections, vmprof.com and PyCharm compliment each other providing deeper insight into your program. With PyCharm you can view the per-line profiling results inside the editor. With the vmprof.com you get a handy visualization of the profiling results as a flame chart and memory usage graph.

Since the PyPy Team runs and maintains the service on vmprof.com (which is by the way free and open-source), I’ll explain some more details here. On vmprof.com you can inspect the generated profile interactively instead of looking at console output. What is sent to vmprof.com? You can find details here.

Flamegraph: Accumulates and displays the most frequent codepaths. It allows you to quickly and accurately identify hot spots in your code. The flame graph below is a very short run of richards.py (Thus it shows a lot of time spent in PyPy's JIT compiler).

List all functions (optionally sorted): the equivalent of the vmprof command line output in the web.

Memory curve: A line plot that shows how how many MBytes have been consumed over the lifetime of your program (see more info in the section below).

Native programs

The new feature introduced in vmprof 0.4.x allows you to look beyond the Python level. As you might know, Python maintains a stack of frames to save the execution. Up to now the vmprof profiles only contained that level of information. But what if you program jumps to native code (such as calling gzip compression on a large file)? Up to now you would not see that information.

Many packages make use of the CPython C API (which we discurage, please lookup cffi for a better way to call C). Have you ever had the issue that you know that your performance problems reach down to, but you could not profile it properly? Now you can!

Let's inspect a very simple Python program to find out why a program is significantly slower on Linux than on Mac:

import numpy as np
n = 1000
a = np.random.random((n, n))
b = np.random.random((n, n))
c = np.dot(np.abs(a), b)

Take two NxN random matrix objects and create a dot product. The first argument to the dot product provides the absolute value of the random matrix.

Run	Python	NumPy	OS	n=...	Took
[1]	CPython 3.5.2	NumPy 1.12.1	Mac OS X, 10.12.3	n=5000	~9 sec
[2]	CPython 3.6.0	NumPy 1.12.1	Linux 64, Kernel 4.9.14	n=1000	~26 sec

Note that the Linux machine operates on a 5 times smaller matrix, still it takes much longer. What is wrong? Is Linux slow? CPython 3.6.0? Well no, lets inspect and [1] and [2] (shown below in that order).

[2] runs on Linux, spends nearly all of the time in PyArray_MatrixProduct2, if you compare to [1] on Mac OS X, you'll see that a lot of time is spent in generating the random numbers and the rest in cblas_matrixproduct.

Blas has a very efficient implementation so you can achieve the same on Linux if you install a blas implementation (such as openblas).

Usually you can spot potential program source locations that take a lot of time and might be the first starting point to resolve performance issues.

Beyond Python programs

It is not unthinkable that the strategy can be reused for native programs. Indeed this can already be done by creating a small cffi wrapper around an entry point of a compiled C program. It would even work for programs compiled from other languages (e.g. C++ or Fortran). The resulting function names are the full symbol name embedded into either the executable symboltable or extracted from the dwarf debugging information. Most of those will be compiler specific and contain some cryptic information.

Memory profiling
We thankfully received a code contribution from the company Blue Yonder. They have built a memory profiler (for Linux and Mac OS X) on top of vmprof.com that displays the memory consumption for the runtime of your process.

You can run it the following way:

$ python -m vmprof --mem --web script.py

By adding --mem, vmprof will capture memory information and display it in the dedicated view on vmprof.com. You can view it by by clicking the 'Memory' switch in the flamegraph view.

There is more

Some more minor highlights contained in 0.4.x:

VMProf support for Windows 64 bit (No native profiling)
VMProf can read profiles generated by another host system
VMProf is now bundled in several binary wheel for fast and easy installation (Mac OS X, Linux 32/64 for CPython 2.7, 3.4, 3.5, 3.6)

Future plans - Profile Streaming

vmprof has not reached the end of development. There are many features we could implement. But there is one feature that could be a great asset to many Python developers.

Continuous delivery of your statistical profile, or in short, profile streaming. One of the great strengths of vmprof is that is consumes very little overhead. It is not a crazy idea to run this in production.

It would require a smart way to stream the profile in the background to vmprof.com and new visualizations to look at much more data your Python service produces.

If that sounds like a solid vmprof improvement, don't hesitate to get in touch with us (e.g. IRC #pypy, mailing list pypy-dev, or comment below)

You can help!

There are some immediate things other people could help with. Either by donating time or money (yes we have occasional contributors which is great)!

We gladly received code contribution for the memory profiler. But it was not enough time to finish the migration completely. Sadly it is a bit brittle right now.
We would like to spend more time on other visualizations. This should include to give a much better user experience on vmprof.com (like a tutorial that explains the visualization that we already have).
Build Windows 32/64 bit wheels (for all CPython versions we currently support)

We are also happy to accept google summer of code projects on vmprof for new visualizations and other improvements. If you qualify and are interested, don't hesitate to ask!

Richard Plangger (plan_rich) and the PyPy Team

[1] Mac OS X https://vmprof.com/#/567aa150-5927-4867-b22d-dbb67ac824ac
[2] Linux64 https://vmprof.com/#/097fded2-b350-4d68-ae93-7956cd10150c

Leysin Winter Sprint Summary

Richard Plangger — Sat, 04 Mar 2017 09:59:00 GMT

Today is the last day of our yearly sprint event in Leysin. We had lots of ideas on how to enhance the current state of PyPy, we went skiing and had interesting discussions around virtual machines, the Python ecosystem, and other real world problems.

Why don't you join us next time?

A usual PyPy sprints day goes through the following stages:

Planning Session: Tasks from previous days that have seen progress or are completed are noted in a shared document. Everyone adds new tasks and then assigns themselves to one or more tasks (usually in pairs). As soon as everybody is happy with their task and has a partner to work with, the planning session is concluded and the work can start.
Discussions: A sprint is a good occasion to discuss difficult and important topics in person. We usually sit down in a separate area in the sprint room and discuss until a) nobody wants to discuss anymore or b) we found a solution to the problem. The good thing is that usally the outcome is b).
Lunch: For lunch we prepare sandwiches and other finger food.
Continue working until dinner, which we eat at a random restaurant in Leysin.
Goto 1 the next day, if sprint has not ended.

Sprints are open to everybody and help newcomers to get started with PyPy (we usually pair you with a developer familiar with PyPy). They are perfect to discuss and find solutions to problems we currently face. If you are eager to join next year, please don't hesitate to register next year around January.

Sprint Summary

Sprint goals included to work on the following topics:

Work towards releasing PyPy 3.5 (it will be released soon)
CPython Extensionsion (CPyExt) modules on PyPy
Have fun in winter sports (a side goal)

Highlights

We have spent lots of time debugging and fixing memory issues on CPyExt. In particular, we fixed a serious memory leak where taking a memoryview would prevent numpy arrays from ever being freed. More work is still required to ensure that our GC always releases arrays in a timely manner.
Fruitful discussions and progress about how to flesh out some details about the unicode representation in PyPy. Our current goal is to use utf-8 as the unicode representation internally and have fast vectorized operations (indexing, check if valid, ...).
PyPy will participate in GSoC 2017 and we will try to allocate more resources to that than last year.
Profile and think about some details how to reduce the starting size of the interpreter. The starting point would be to look at the parser and reduce the amount of strings to keep alive.
Found a topic for a student's master thesis: correctly freeing cpyext reference cycles.
Run lots of Python3 code on top of PyPy3 and resolve issues we found along the way.
Initial work on making RPython thread-safe without a GIL.

List of attendees

- Stefan Beyer

- Antonio Cuni

- Maciej Fijalkowski

- Manuel Jacob

- Ronan Lamy

- Remi Meier

- Richard Plangger

- Armin Rigo

- Robert Zaremba

We would like to thank our donors for the continued support of the PyPy project and we looking forward to next years sprint in Leysin.

The PyPy Team

Vectorization extended. PowerPC and s390x

Richard Plangger — Thu, 03 Nov 2016 08:35:00 GMT

We are happy to announce that JIT support in both the PowerPC backend and the
s390x backend have been enhanced. Both can now vectorize loops via SIMD
instructions. Special thanks to IBM for funding this work.

If you are not familiar with this topic you can read more details here.

There are many more enhancements under the hood. Most notably, all pure operations are now delayed until the latest possible point. In some cases indices have been calculated more than once or they needed an additional register, because the old value is still used. Additionally it is now possible to load quadword-aligned memory in both PPC and s390x (x86 currently cannot do that).

NumPy & CPyExt

The community and core developers have been moving CPyExt towards a complete, but emulated, layer for CPython C extensions. This is great, because the one restriction preventing the wider deployment of PyPy in several scenarios will hopefully be removed. However, we advocate not to use CPyExt, but rather to not write C code at all (let PyPy speed up your Python code) or use cffi.

The work done here to support vectorization helps micronumpy (NumPyPy) to speed up operations for PPC and s390x. So why is PyPy supporting both NumPyPy and NumPy, do we actually need both? Yes, there are places where gcc can beat the JIT, and places where the tight integration between NumPyPy and PyPy is more performant. We do have plans to integrate both, hijacking the C-extension method calls to use NumPyPy where we know NumPyPy can be faster.

Just to give you an idea why this is a benefit:

NumPy arrays can carry custom dtypes and apply user defined python functions on the arrays. How could one optimize this kind of scenario? In a traditional setup, you cannot. But as soon as NumPyPy is turned on, you can suddenly JIT compile this code and vectorize it.

Another example is element access that occurs frequently, or any other calls that cross between Python and the C level frequently.

Benchmarks

Let's have a look at some benchmarks reusing mikefc's numpy benchmark suite (find the forked version here). I only ran a subset of microbenchmarks, showing that the core functionality is
functioning properly. Additionally it has been rewritten to use perf instead of the timeit stdlib module.

Setup

x86 runs on a Intel i7-2600 clocked at 3.40GHz using 4 cores. PowerPC runs on the Power 8 clocked at 3.425GHz providing 160 cores. Last but not least the mainframe machine clocked up to 4 GHz, but fully virtualized (as it is common for such machines). Note that PowerPC is a non private remote machine. It is used by many users and it is crowded with processes. It is hard to extract a stable benchmark there.

x86 ran on Fedora 24 (kernel version of 4.8.4), PPC ran on Fedora 21 (kernel version 3.17.4) and s390x ran on Redhat Linux 7.2 (kernel version 3.10.0). Respectivley, numpy on cpython had openblas available on x86, no blas implementation were present on s390x and PPC provided blas and lapack.

As you can see all machines run very different configurations. It does not make sense to compare across platforms, but rather implementations on the same platform.

Blue shows CPython 2.7.10+ available on that platform using the latest NumPy (1.11). Micro NumPy is used for PyPy. PyPy+ indicates that the vectorization optimization is turned on.
All bar charts show the median value of all runs (5 samples, 100 loops, 10 inner loops, for the operations on vectors (not matrices) the loops are set to 1000). PyPy additionally gets 3 extra executions to warmup the JIT.

The comparison is really comparing speed of machine code. It compares the PyPy's JIT output vs GCC's output. It has little to do with the speed of the interpreter.

Both new SIMD backends speedup the numeric kernels. Some times it is near to the speed of CPython, some times it is faster. The maximum parallelism very much depends on the extension emitted by the compiler. All three SIMD backends have the same vector register size (which is 128 bit). This means that all three behave similar but ppc and s390x gain more because they can load 128bit of memory from quadword aligned memory.

Future directions

Python is achieving rapid adoption in data science. This is currently a trend emerging in Europe, and Python is already heavily used for data science in the USA many other places around the world.

PyPy can make a valuable contribution for data scientists, helping them to rapidly write scientific programs in Python and run them at near native speed. If you happen to be in that situation, we are eager to hear you feedback or resolve your issues and also work together to improve the performance of your,
code. Just get in touch!

Richard Plangger (plan_rich) and the PyPy team

PyPy Tooling Upgrade: JitViewer and VMProf

Richard Plangger — Thu, 11 Aug 2016 11:52:00 GMT

We are happy to announce a major JitViewer (JV) update.
JV allows you to inspect RPython's internal compiler representation (the language in which PyPy is implemented) including the generated machine code of your program. It can graphically show you details of the JIT compiled code and helps you pinpoint issues in your program.

VMProf is a statistical CPU profiler for python imposing very little overhead at runtime.

Both VMProf and JitViewer share a common goal: Present useful information for your python program.
The combination of both can reveal more information than either alone.
That is the reason why they are now both packaged together.
We also updated vmprof.com with various bug fixes and changes including an all new interface to JV.

This work was done with the goal of improving tooling and libraries around the Python/PyPy/RPython ecosystem.
Some of the tools we have developed:

CFFI - Foreign Function Interface that avoids CPyExt (CFFI docs)
RevDB - A reverse debugger for python (RevDB blog post)

and of course the tools we discuss here:

VMProf - A statistical CPU profiler (VMProf docs)
JitViewer - Visualization of the log file produced by RPython (JitLog docs)

A "brand new" JitViewer

JitViewer has two pieces: you create a log file when running your program, and then use a graphic tool to view what happened.

The old logging format was a hard-to-maintain, plain-text-logging facility. Frequent changes often broke internal tools.
Additionally, the logging output of a long running program required a lot of disk space.

Our new binary format encodes data densely, makes use of some compression (gzip), and tries to remove repetition where possible.
It also supports versioning for future proofing and can be extended easily.

And *drumroll* you no longer need to install a tool to view the log yourself
anymore! The whole system moved to vmprof.com and you can use it any time.

Sounds great. But what can you do with it? Here are two examples for a PyPy user:

PyPy crashed? Did you discover a bug?

For some hard to find bugs it is often necessary to look at the compiled code. The old
procedure often required you to upload a plain text file which was hard to parse and to look through.

A better way to share a crash report is to install the ``vmprof`` module from PyPi and execute either of the two commands:

# this program does not crash, but has some weird behaviour
$ pypy -m jitlog --web <your program args>
...
PyPy Jitlog: https://vmprof.com/#/<hash>/traces
# this program segfaults
$ pypy -m jitlog -o /tmp/log <your program args>
...
<Segfault>
$ pypy -m jitlog --upload /tmp/log
PyPy Jitlog: https://vmprof.com/#/<hash>/traces

Providing the link in the bug report allows PyPy developers to browse and identify potential issues.

Speed issues

VMProf is a great tool to find hot spots that consume a lot of time in your program. As soon as you have identified code that runs slowly, you can switch to jitlog and maybe pinpoint certain aspects that do not behave as expected. You will find an overview, and are able to browse the generated code. If you cannot make sense of all that, you can just share the link with us and we can have a look too.

Future direction

We hope that the new release will help both PyPy developers and PyPy users resolve potential issues and easily point them out.

Here are a few ideas what might come in the next few releases:

Combination of CPU profiles and the JITLOG (sadly did not make it into the current release).
Extend vmprof.com to be able to query vmprof/jitlog.
An example query for vmprof: 'methods.callsites() > 5' and
for the jitlog would be 'traces.contains('call_assembler').hasbridge('*my_func_name*')'.
Extend the jitlog to capture the information of the optimization stage.

Richard Plangger (plan_rich) and the PyPy team

PyPy Enterprise Edition

Richard Plangger — Mon, 18 Apr 2016 10:13:00 GMT

With the latest additions, PyPy's JIT now supports the Z architecture on Linux. The newest architecture revision (also known as s390x, or colloquially referred to as "big iron") is the 64-bit extension for IBM mainframes. Currently only Linux 64 bit is supported (not z/OS nor TPF).
This is the fourth assembler backend supported by PyPy in addition to x86 (32 and 64), ARM (32-bit only) and PPC64 (both little- and big-endian). It might seem that we kind of get a hang of new architectures. Thanks to IBM for funding this work!

History

When I went to university one lecture covered the prediction of Thomas Watson in 1943. His famous quote "I think there is a world market for maybe five computers ...", turned out not to be true.

However, even 70 years later, mainframes are used more often than you think. They back critical tasks requiring a high level of stability/security and offer high hardware and computational utilization rates by virtualization.

With the new PyPy JIT backend we are happy to present a fast Python virtual machine for mainframes and contribute more free software running on s390x.

Meta tracing

Even though the JIT backend has been tested on PyPy, it is not restricted to the Python programming language. Do you have a great idea for a DSL, or another language that should run on mainframes? Go ahead and just implement your interpreter using RPython.

How do I get a copy?

PyPy can be built using the usual instructions found here. As soon as the next PyPy version has been released we will provide binaries. Until then you can just grab a nightly here.We are currently busy to get the next version of PyPy ready, so an official release will be rolled out soon.

Comparing s390x to x86

The goal of this comparison is not to scientifically evaluate the benefits/disadvantages on s390x, but rather to see that PyPy's architecture delivers the same benefits as it does on other platforms. Similar to the comparison done for PPC I ran the benchmarks using the same setup. The first column is the speedup of the PyPy JIT VM compared to the speedup of a pure PyPy interpreter 1). Note that the s390x's OS was virtualized.

Label               x86     s390x      s390x (run 2)

ai                 13.7      12.4       11.9
bm_chameleon        8.5       6.3        6.8
bm_dulwich_log      5.1       5.0        5.1
bm_krakatau         5.5       2.0        2.0
bm_mako             8.4       5.8        5.9
bm_mdp              2.0       3.8        3.8
chaos              56.9      52.6       53.4
crypto_pyaes       62.5      64.2       64.2
deltablue           3.3       3.9        3.6
django             28.8      22.6       21.7
eparse              2.3       2.5        2.6
fannkuch            9.1       9.9       10.1
float              13.8      12.8       13.8
genshi_text        16.4      10.5       10.9
genshi_xml          8.2       7.9        8.2
go                  6.7       6.2       11.2
hexiom2            24.3      23.8       23.5
html5lib            5.4       5.8        5.7
json_bench         28.8      27.8       28.1
meteor-contest      5.1       4.2        4.4
nbody_modified     20.6      19.3       19.4
pidigits            1.0      -1.1       -1.0
pyflate-fast        9.0       8.7        8.5
pypy_interp         3.3     4.2        4.4
raytrace-simple    69.0     100.9       93.4
richards           94.1      96.6       84.3
rietveld            3.2       2.5        2.7
slowspitfire        2.8       3.3        4.2
spambayes           5.0       4.8        4.8
spectral-norm      41.9      39.8       42.6
spitfire            3.8       3.9        4.3
spitfire_cstringio 7.6       7.9        8.2
sympy_expand        2.9       1.8        1.8
sympy_integrate     4.3       3.9        4.0
sympy_str           1.5       1.3        1.3
sympy_sum           6.2       5.8        5.9
telco              61.2      48.5       54.8
twisted_iteration 55.5      41.9       43.8
twisted_names       8.2       9.3        9.7
twisted_pb         12.1      10.4       10.2
twisted_tcp         4.9       4.8        5.2

Geometric mean:    9.31      9.10       9.43

As you can see the benefits are comparable on both platforms.
Of course this is scientifically not good enough, but it shows a tendency. s390x can achieve the same results as you can get on x86.

Are you running your business application on a mainframe? We would love to get some feedback. Join us in IRC tell us if PyPy made your application faster!

plan_rich & the PyPy Team

1) PyPy revision for the benchmarks: 4b386bcfee54

Automatic SIMD vectorization support in PyPy

Richard Plangger — Tue, 20 Oct 2015 14:38:00 GMT

Hi everyone,

it took some time to catch up with the JIT refacrtorings merged in this summer. But, (drums) we are happy to announce that:

The next release of PyPy, "PyPy 4.0.0", will ship the new auto vectorizer

The goal of this project was to increase the speed of numerical applications in both the NumPyPy library and for arbitrary Python programs. In PyPy we have focused a lot on improvements in the 'typical python workload', which usually involves object and string manipulations, mostly for web development. We're hoping with this work that we'll continue improving the other very important Python use case - numerics.

What it can do!

It targets numerics only. It will not execute object manipulations faster, but it is capable of enhancing common vector and matrix operations.
Good news is that it is not specifically targeted for the NumPy library and the PyPy virtual machine. Any interpreter (written in RPython) is able make use of the vectorization. For more information about that take a look here, or consult the documentation. For the time being it is not turn on by default, so be sure to enable it by specifying --jit vec=1 before running your program.

If your language (written in RPython) contains many array/matrix operations, you can easily integrate the optimization by adding the parameter 'vec=1' to the JitDriver.

NumPyPy Improvements

Let's take a look at the core functions of the NumPyPy library (*).
The following tests tests show the speedup of the core functions commonly used in Python code interfacing with NumPy, on CPython with NumPy, on the PyPy 2.6.1 relased several weeks ago, and on PyPy 15.11 to be released soon. Timeit was used to test the time needed to run the operation in the plot title on various vector (lower case) and square matrix (upper case) sizes displayed on the X axis. The Y axis shows the speedup compared to CPython 2.7.10. This means that higher is better.

In comparison to PyPy 2.6.1, the speedup greatly improved. The hardware support really strips down the runtime of the vector and matrix operations. There is another operation we would like to highlight: the dot product.
It is a very common operation in numerics and PyPy now (given a moderate sized matrix and vector) decreases the time spent in that operation. See for yourself:

These are nice improvements in the NumPyPy library and we got to a competitive level only making use of SSE4.1.

Future work

This is not the end of the road. The GSoC project showed that it is possible to implement this optimization in PyPy. There might be other improvements we can make to carry this further:

Check alignment at runtime to increase the memory throughput of the CPU
Support the AVX vector extension which (at least) doubles the size of the vector register
Handle each and every corner case in Python traces to enable it globally
Do not rely only on loading operations to trigger the analysis, there might be cases where combination of floating point values could be done in parallel

Cheers,
The PyPy Team