PyPy (Posts about pypy3)

Leysin 2020 Sprint Report

hodgestar — Tue, 17 Mar 2020 21:57:00 GMT

At the end of February ten of us gathered in Leysin, Switzerland to work on
a variety of topics including HPy, PyPy Python 3.7 support and the PyPy
migration to Heptapod.

We had a fun and productive week. The snow was beautiful. There was skiing
and lunch at the top of Berneuse, cooking together, some late nights at
the pub next door, some even later nights coding, and of course the
obligatory cheese fondue outing.

There were a few of us participating in a PyPy sprint for the first time
and a few familiar faces who had attended many sprints. Many different
projects were represented including PyPy, HPy, GraalPython,
Heptapod, and rust-cpython. The atmosphere was relaxed and welcoming, so if
you're thinking of attending the next one -- please do!

Topics worked on:

HPy

HPy is a new project to design and implement a better API for extending
Python in C. If you're unfamiliar with it you can read more about it at
HPy.

A lot of attention was devoted to the Big HPy Design Discussion which
took up two full mornings. So much was decided that this will likely
get its own detailed write-up, but bigger topics included:

the HPy GetAttr, SetAttr, GetItem and SetItem methods,
HPy_FromVoidP and HPy_AsVoidP for passing HPy handles to C functions
that pass void* pointers to callbacks,
avoiding having va_args as part of the ABI,
exception handling,
support for creating custom types.

Quite a few things got worked on too:

implemented support for writing methods that take keyword arguments with
HPy_METH_KEYWORDS,
implemented HPy_GetAttr, HPy_SetAttr, HPy_GetItem, and HPy_SetItem,
started implementing support for adding custom types,
started implementing dumping JSON objects in ultrajson-hpy,
refactored the PyPy GIL to improve the interaction between HPy and
PyPy's cpyext,
experimented with adding HPy support to rust-cpython.

And there was some discussion of the next steps of the HPy initiative
including writing documentation, setting up websites and funding, and
possibly organising another HPy gathering later in the year.

PyPy

Georges gave a presentation on the Heptapod topic and branch workflows
and showed everyone how to use hg-evolve.
Work was done on improving the PyPy CI buildbot post the move to
heptapod, including a light-weight pre-merge CI and restricting
when the full CI is run to only branch commits.
A lot of work was done improving the -D tests.

Miscellaneous

Armin demoed VRSketch and NaN Industries in VR, including an implementation
of the Game of Life within NaN Industries!
Skiing!

Aftermath

Immediately after the sprint large parts of Europe and the world were
hit by the COVID-19 epidemic. It was good to spend time together before
travelling ceased to be a sensible idea and many gatherings were cancelled.

Keep safe out there everyone.

The HPy & PyPy Team & Friends

In joke for those who attended the sprint: Please don't replace this blog post
with its Swedish translation (or indeed a translation to any other language :).

PyPy3 2.3.1 - Fulcrum

Philip Jenvey — Fri, 20 Jun 2014 21:31:00 GMT

We're pleased to announce the first stable release of PyPy3. PyPy3
targets Python 3 (3.2.5) compatibility.

We would like to thank all of the people who donated to the py3k proposal
for supporting the work that went into this.

You can download the PyPy3 2.3.1 release here:

https://pypy.org/download.html#pypy3-2-3-1

Highlights

The first stable release of PyPy3: support for Python 3!
The stdlib has been updated to Python 3.2.5
Additional support for the u'unicode' syntax (PEP 414) from Python 3.3
Updates from the default branch, such as incremental GC and various JIT
improvements
Resolved some notable JIT performance regressions from PyPy2:

Re-enabled the previously disabled collection (list/dict/set) strategies

Resolved performance of iteration over range objects

Resolved handling of Python 3's exception __context__ unnecessarily forcing
frame object overhead

What is PyPy?

PyPy is a very compliant Python interpreter, almost a drop-in replacement for
CPython 2.7.6 or 3.2.5. It's fast due to its integrated tracing JIT compiler.

This release supports x86 machines running Linux 32/64, Mac OS X 64, Windows,
and OpenBSD,
as well as newer ARM hardware (ARMv6 or ARMv7, with VFPv3) running Linux.

While we support 32 bit python on Windows, work on the native Windows 64
bit python is still stalling, we would welcome a volunteer
to handle that.

How to use PyPy?

We suggest using PyPy from a virtualenv. Once you have a virtualenv
installed, you can follow instructions from pypy documentation on how
to proceed. This document also covers other installation schemes.

Cheers,
the PyPy team

Py3k status update #13

Philip Jenvey — Tue, 18 Feb 2014 02:33:00 GMT

This is the 13th status update about our work on the py3k branch, which we
can work on thanks to all of the people who donated to the py3k proposal.

We're just finishing up a cleanup of int/long types. This work helps the py3k
branch unify these types into the Python 3 int and restore JIT compilation of
machine sized integers.

This cleanup also removes multimethods from these types. PyPy has
historically used a clever implementation of multimethod dispatch for declaring
methods of the __builtin__ types in RPython.

This multimethod scheme provides some convenient features for doing this,
however we've come to the conclusion that it may be more trouble than it's
worth. A major problem of multimethods is that they generate a large amount of
stub methods which burden the already lengthy and memory hungry RPython
translation process. Also, their implementation and behavior can be somewhat
complicated/obscure.

The alternative to multimethods involves doing the work of the type checking
and dispatching rules in a more verbose, manual way. It's a little more work in
the end but less magical.

Recently, Manuel Jacob finished a large cleanup effort of the
unicode/string/bytearray types that also removed their multimethods. This work
also benefits the py3k branch: it'll help with future PEP 393 (or PEP 393
alternative) work. This effort was partly sponsored by Google's Summer of
Code: thanks Manuel and Google!

Now there's only a couple major pieces left in the multimethod removal (the
float/complex types and special marshaling code) and a few minor pieces that
should be relatively easy.

In conclusion, there's been some good progress made on py3k and multimethod
removal this winter, albeit a bit slower than we would have liked.

cheers,
Phil

Py3k status update #12

Philip Jenvey — Tue, 12 Nov 2013 23:16:00 GMT

This is the 12th status update about our work on the py3k branch, which we
can work on thanks to all of the people who donated to the py3k proposal.

Here's an update on the recent progress:

Thank you to everyone who has provided initial feedback on the PyPy3 2.1 beta
1 release. We've gotten a number of bug reports, most of which have been
fixed.
As usual, we're continually keeping up with changes from the default
branch. Oftentimes these merges come at a cost (conflicts and or
reintegration of py3k changes) but occasionally we get goodies for free, such
as the recent JIT optimizations and incremental garbage collection.
We've been focusing on re-optimizing Python 2 int sized (machine sized)
integers:

We have a couple of known, notable speed regressions in the PyPy3 beta release
vs regular PyPy. The major one being with Python 2.x int sized (or machine
sized) integers.

Python 3 drops the distinction between int and long types. CPython 3.x
accomplishes this by removing the old int type entirely and renaming the long
type to int. Initially, we've done the same for PyPy3 for the sake of
simplicity and getting everything working.

However PyPy's JIT is capable of heavily optimizing these machine sized integer
operations, so this came with a regression in performance in this area.

We're now in the process of solving this. Part of this work also involves some
house cleaning on these numeric types which also benefits the default branch.

cheers,
Phil

Py3k status update #11

Philip Jenvey — Wed, 12 Jun 2013 19:17:00 GMT

This is the 11th status update about our work on the py3k branch, which we
can work on thanks to all of the people who donated to the py3k proposal.

Here's some highlights of the progress made since the previous update:

PyPy py3k now matches CPython 3's hash code for
int/float/complex/Decimal/Fraction
Various outstanding unicode identifier related issues were
resolved. E.g. test_importlib/pep263/ucn/unicode all now fully pass. Various
usage of identifiers (in particular type and module names) have been fixed to
handle non-ascii names -- mostly around display of reprs and exception
messages.
The unicodedata database has been upgraded to 6.0.0.
Windows support has greatly improved, though it could still use some more
help (but so does the default branch to a certain degree).
Probably the last of the parsing related bugs/features have been taken care
of.
Of course various other smaller miscellaneous fixes

This leaves the branch w/ only about 5 outstanding failures of the stdlib test
suite:

test_float
1 failing test about containment of floats in collections.
test_memoryview
Various failures: requires some bytes/str changes among other things (Manuel
Jacob's has some progress on this on the py3k-memoryview branch)
test_multiprocessing
1 or more tests deadlock on some platforms
test_sys and test_threading
2 failing tests for the New GIL's new API

Probably the biggest feature left to tackle is the New GIL.

We're now pretty close to pushing an initial release. We had planned for one
around PyCon, but having missed that we've put some more effort into the branch
to provide a more fully-fledged initial release.

Thanks to the following for their contributions: Manuel Jacob, Amaury Forgeot
d'Arc, Karl Ramm, Jason Chu and Christian Hudon.

cheers,
Phil

Py3k status update #10

Philip Jenvey — Tue, 05 Mar 2013 20:00:00 GMT

This is the tenth status update about our work on the py3k branch, which we
can work on thanks to all of the people who donated to the py3k proposal.

There's been significant progress since the last update: the linux x86-32
buildbot now passes 289 out of approximately 354 modules (with 39 skips) of
CPython's regression test suite.

That means there's only 26 test module failures left! The list of major items
remaining for 3.2 compatibility are now short enough to list here, with their
related tests:

Tokenizer support for non-ascii identifiers

test_importlib

test_pep263

memoryview (Manuel Jacob's tackling this on the py3k-memoryview branch)

test_memoryview

multiprocessing module currently deadlocks

test_multiprocessing

Buggy handling of the new extended unpacking syntax by the compiler:

test_unpack_ex

The new Global Interpreter Lock and new thread signal handling

test_threading

test_threadsignals

test_sys

Upgrade unicodedata to 6.0.0 (requires updates to the actual unicodedata
generation script)

test_ucn

test_unicode

test_unicodedata

CPyExt

test_capi (currently crashes)

Update int's hash code to match to CPython (float's is already updated on the
py3k-newhash branch. note that PyPy 2.x doesn't even totally match
CPython's hashing)

test_decimal

test_fractions

test_numeric_tower

Miscellaneous:

test_complex

test_float

test_peepholer

test_range

test_sqlite (a new cffi based version seems to be coming)

test_ssl

test_struct

test_subprocess

test_sys_settrace

test_time

Additionally there are still a number of failures in PyPy's internal test
suite. These tests are usually ran against untranslated versions of PyPy during
development. However we've now began running them against a fully translated
version of PyPy on the buildbot too (thanks to Amaury for setting this
up). This further ensures that our tests and implementation are sane.

We're getting closer to producing an initial alpha release. Before that happens
we'd like to see:

further test fixes
the results of test runs on other major platforms (e.g. linux x86-64 and osx
seem to have some additional failures as of now)
some basic real world testing

Finally I'd like to thank Manuel Jacob for his various contributions over the
past month, including fixing the array and ctypes modules among other things,
and also Amaury Forgeot d'Arc for his ongoing excellent contributions.

cheers,
Phil

Py3k status update #9

Philip Jenvey — Thu, 10 Jan 2013 06:04:00 GMT

This is the ninth status update about our work on the py3k branch, which
we can work on thanks to all of the people who donated to the py3k
proposal.

Just a very short update on December's work: we're now passing about 223 of
approximately 355 modules of CPython's regression test suite, up from passing
194 last month.

Some brief highlights:

More encoding related issues were addressed. e.g. now most if not all the
multibytecodec test modules pass.
Fixed some path handling issues (test_os, test_ntpath and
test_posixpath now pass)
We now pass test_class, test_descr and almost test_builtin (among
other things): these are notable as they are fairly extensive test suites of
core aspects of the langauge.
Amaury Forgeot d'Arc continued making progress on CPyExt (thanks again!)

cheers,
Phil

Py3k status update #8

Philip Jenvey — Tue, 04 Dec 2012 22:30:00 GMT

This is the eight status update about our work on the py3k branch, which
we can work on thanks to all of the people who donated to the py3k
proposal.

Just a short update on November's work: we're now passing about 194 of
approximately 355 modules of CPython's regression test suite, up from passing
160 last month. Many test modules only fail a small number of individual tests
now.

We'd like to thank Amaury Forgeot d'Arc for his contributions, in particular he
has made significant progress on updating CPyExt for Python 3 this month.

Some other highlights:

test_marshal now passes, and there's been significant progress on
pickling (thanks Kenny Levinsen and Amaury for implementing
int.{to,from}_bytes)
We now have a _posixsubprocess module
More encoding related fixes, which affects many failing tests
_sre was updated and now test_re almost passes
Exception behavior is almost complete per the Python 3 specs, what's mostly
missing now are the new __context__ and __traceback__ attributes (PEP
3134)
Fixed some crashes and deadlocks occurring during the regression tests
We merged the unicode-strategies branch both to default and to py3k: now we
have versions of lists, dictionaries and sets specialized for unicode
elements, as we already had for strings.
However, for string-specialized containers are still faster in some cases
because there are shortcuts which have not been implemented for unicode yet
(e.g., constructing a set of strings from a list of strings). The plan is to
completely kill the shortcuts and improve the JIT to produce the fast
version automatically for both the string and unicode versions, to have a
more maintainable codebase without sacrificing the speed. The autoreds
branch (already merged) was a first step in this direction.

cheers,
Philip&Antonio

Py3k status update #7

Philip Jenvey — Fri, 02 Nov 2012 15:47:00 GMT

This is the seventh status update about our work on the py3k branch, which
we can work on thanks to all of the people who donated to the py3k
proposal.

The biggest news is that this month Philip started to work on py3k in parallel
to Antonio. As such, there was an increased amount of activity.

The py3k buildbots now fully translate the branch every night and run the
Python standard library tests.

We currently pass 160 out of approximately 355 modules of CPython's standard
test suite, fail 144 and skip approximately 51.

Some highlights:

dictviews (the objects returned by dict.keys/values/items) has been greatly
improved, and now they full support set operators
a lot of tests has been fixed wrt complex numbers (and in particular the
__complex__ method)
_csv has been fixed and now it correctly handles unicode instead of bytes
more parser fixes, py3k list comprehension semantics; now you can no longer
access the list comprehension variable after it finishes
2to3'd most of the lib_pypy modules (pypy's custom standard lib
replacements/additions)
py3-enabled pyrepl: this means that finally readline works at the command
prompt, as well as builtins.input(). pdb seems to work, as well as
fancycompleter to get colorful TAB completions :-)
py3 round
further tightening/cleanup of the unicode handling (more usage of
surrogateescape, surrogatepass among other things)
as well as keeping up with some big changes happening on the default branch
and of course various other fixes.

Finally, we would like to thank Amaury Forgeot d'Arc for his significant
contributions.

cheers,
Philip&Antonio

Py3k status update #6

Antonio Cuni — Wed, 26 Sep 2012 09:50:00 GMT

This is the sixth status update about our work on the py3k branch, which we
can work on thanks to all of the people who donated to the py3k proposal.

The coolest news is not about what we did in the past weeks, but what we will
do in the next: I am pleased to announce that Philip Jenvey has been
selected by the PyPy communitiy to be funded for his upcoming work on py3k,
thanks to your generous donations. He will start to work on it shortly, and he
will surely help the branch to make faster progress. I am also particularly
happy of this because Philip is the first non-core developer who is getting
paid with donations: he demonstrated over the past months to be able to work
effectively on PyPy, and so we were happy to approve his application for the
job. This means that anyone can potentially be selected in the future, the
only strict requirement is to have a deep interest in working on PyPy and to
prove to be able to do so by contributing to the project.

Back to the status of the branch. Most of the work since the last status
update has been done in the area of, guess what? Unicode strings. As usual,
this is one of the most important changes between Python 2 and Python 3, so
it's not surprising. The biggest news is that now PyPy internally supports
unicode identifiers (such as names of variables, functions, attributes, etc.),
whereas earlier it supported only ASCII bytes strings. The changes is still
barely visible from the outside, because the parser still rejects non-ASCII
identifiers, however you can see it with a bit of creativity:

>>>> def foo(x): pass
>>>> foo(**{'àèìòù': 42})
Traceback (most recent call last):
  File "<console>", line 1, in <module>
TypeError: foo() got an unexpected keyword argument 'àèìòù'

Before the latest changes, you used to get question marks instead of the
proper name for the keyword argument. Although this might seem like a small
detail, it is a big step towards a proper working Python 3 interpreter and it
required a couple of days of headaches. A spin-off of this work is that now
RPython has better built-in support for unicode (also in the default branch):
for example, it now supports unicode string formatting (using the percent
operator) and the methods .encode/.decode('utf-8').

Other than that there is the usual list of smaller issues and bugs that got
fixed, including (but not limited to):

teach the compiler when to emit the new opcode DELETE_DEREF (and
implement it!)

detect when we use spaces and TABs inconsistently in the source code, as
CPython does

fix yet another bug related to the new lexically scoped exceptions (this
is the last one, hopefully)

port some of the changes that we did to the standard CPython 2.7 tests to
3.2, to mark those which are implementation details and should not be run on
PyPy

Finally, I would like to thank Amaury Forgeot d'Arc and Ariel Ben-Yehuda for
their work on the branch; among other things, Amaury recently worked on
cpyext and on the PyPy _cffi_backend, while Ariel submitted a patch to
implement PEP 3138.