PyPy (Posts by Philip Jenvey)

PyPy3.3 v5.2 alpha 1 released

Philip Jenvey — Mon, 30 May 2016 22:53:00 GMT

We're pleased to announce the first alpha release of PyPy3.3 v5.2. This is the
first release of PyPy which targets Python 3.3 (3.3.5) compatibility.

We would like to thank all of the people who donated to the py3k proposal
for supporting the work that went into this and future releases.

You can download the PyPy3.3 v5.2 alpha 1 release here:

https://pypy.org/download.html#python-3-3-5-compatible-pypy3-3-v5-2

Highlights

Python 3.3.5 support!
- Being an early alpha release, there are some missing features such as a
  PEP 393-like space efficient string representation and known issues
  including performance issues (e.g. issue #2305). The focus for this
  release has been updating to 3.3 compatibility. Windows is also not yet
  supported.
ensurepip is also included (it's only included in CPython 3 >= 3.4).

What is PyPy?

PyPy is a very compliant Python interpreter, almost a drop-in replacement for
CPython 2.7.10 and one day 3.3.5. It's fast due to its integrated tracing JIT
compiler.

We also welcome developers of other dynamic languages to see what RPython
can do for them.

This release supports:

x86 machines on most common operating systems except Windows
(Linux 32/64, Mac OS X 64, OpenBSD, FreeBSD),

newer ARM hardware (ARMv6 or ARMv7, with VFPv3) running Linux,

big- and little-endian variants of PPC64 running Linux,

s390x running Linux

Please try it out and let us know what you think. We welcome feedback, we know
you are using PyPy, please tell us about it!

We'd especially like to thank these people for their contributions to this
release:

Manuel Jacob, Ronan Lamy, Mark Young, Amaury Forgeot d'Arc, Philip Jenvey,
Martin Matusiak, Vasily Kuznetsov, Matti Picus, Armin Rigo and many others.

Cheers

The PyPy Team

PyPy3 2.4.0 released

Philip Jenvey — Tue, 21 Oct 2014 18:02:00 GMT

We're pleased to announce the availability of PyPy3 2.4.0!

This release contains several bugfixes and enhancements. Among the user-facing improvements specific to PyPy3:

Better Windows compatibility, e.g. the nt module functions _getfinalpathname & _getfileinformation are now supported (the former is required for the popular pathlib library for example)
Various fsencode PEP 383 related fixes to the posix module (readlink, uname, ttyname and ctermid) and improved locale handling
Switched the default binary name on POSIX distributions from 'pypy' to 'pypy3' (which symlinks to to 'pypy3.2')
Fixed a couple different crashes related to parsing Python 3 source code

And improvements shared with the recent PyPy 2.4.0 release:

internal refactoring in string and GIL handling which led to significant speedups
improved handling of multiple objects (like sockets) in long-running programs. They are collected and released more efficiently, reducing memory use. In simpler terms - we closed what looked like a memory leak
Windows builds now link statically to zlib, expat, bzip, and openssl-1.0.1i
Many issues were resolved since the 2.3.1 release in June

You can download PyPy3 2.4.0 here https://pypy.org/download.html.

PyPy is a very compliant Python interpreter, almost a drop-in replacement for CPython 2.7 and 3.2.5. It's fast (pypy 2.4 and cpython 2.7.x performance comparison) due to its integrated tracing JIT compiler.

This release supports x86 machines running Linux 32/64, Mac OS X 64, Windows, and OpenBSD, as well as newer ARM hardware (ARMv6 or ARMv7, with VFPv3) running Linux.
We would like to thank our donors for the continued support of the PyPy project.

The complete release notice is here.

Please try it out and let us know what you think. We especially welcome success stories, please tell us about how it has helped you!

Cheers, The PyPy Team

PyPy3 2.3.1 - Fulcrum

Philip Jenvey — Fri, 20 Jun 2014 21:31:00 GMT

We're pleased to announce the first stable release of PyPy3. PyPy3
targets Python 3 (3.2.5) compatibility.

We would like to thank all of the people who donated to the py3k proposal
for supporting the work that went into this.

You can download the PyPy3 2.3.1 release here:

https://pypy.org/download.html#pypy3-2-3-1

Highlights

The first stable release of PyPy3: support for Python 3!
The stdlib has been updated to Python 3.2.5
Additional support for the u'unicode' syntax (PEP 414) from Python 3.3
Updates from the default branch, such as incremental GC and various JIT
improvements
Resolved some notable JIT performance regressions from PyPy2:

Re-enabled the previously disabled collection (list/dict/set) strategies

Resolved performance of iteration over range objects

Resolved handling of Python 3's exception __context__ unnecessarily forcing
frame object overhead

What is PyPy?

PyPy is a very compliant Python interpreter, almost a drop-in replacement for
CPython 2.7.6 or 3.2.5. It's fast due to its integrated tracing JIT compiler.

This release supports x86 machines running Linux 32/64, Mac OS X 64, Windows,
and OpenBSD,
as well as newer ARM hardware (ARMv6 or ARMv7, with VFPv3) running Linux.

While we support 32 bit python on Windows, work on the native Windows 64
bit python is still stalling, we would welcome a volunteer
to handle that.

How to use PyPy?

We suggest using PyPy from a virtualenv. Once you have a virtualenv
installed, you can follow instructions from pypy documentation on how
to proceed. This document also covers other installation schemes.

Cheers,
the PyPy team

Py3k status update #13

Philip Jenvey — Tue, 18 Feb 2014 02:33:00 GMT

This is the 13th status update about our work on the py3k branch, which we
can work on thanks to all of the people who donated to the py3k proposal.

We're just finishing up a cleanup of int/long types. This work helps the py3k
branch unify these types into the Python 3 int and restore JIT compilation of
machine sized integers.

This cleanup also removes multimethods from these types. PyPy has
historically used a clever implementation of multimethod dispatch for declaring
methods of the __builtin__ types in RPython.

This multimethod scheme provides some convenient features for doing this,
however we've come to the conclusion that it may be more trouble than it's
worth. A major problem of multimethods is that they generate a large amount of
stub methods which burden the already lengthy and memory hungry RPython
translation process. Also, their implementation and behavior can be somewhat
complicated/obscure.

The alternative to multimethods involves doing the work of the type checking
and dispatching rules in a more verbose, manual way. It's a little more work in
the end but less magical.

Recently, Manuel Jacob finished a large cleanup effort of the
unicode/string/bytearray types that also removed their multimethods. This work
also benefits the py3k branch: it'll help with future PEP 393 (or PEP 393
alternative) work. This effort was partly sponsored by Google's Summer of
Code: thanks Manuel and Google!

Now there's only a couple major pieces left in the multimethod removal (the
float/complex types and special marshaling code) and a few minor pieces that
should be relatively easy.

In conclusion, there's been some good progress made on py3k and multimethod
removal this winter, albeit a bit slower than we would have liked.

cheers,
Phil

Py3k status update #12

Philip Jenvey — Tue, 12 Nov 2013 23:16:00 GMT

This is the 12th status update about our work on the py3k branch, which we
can work on thanks to all of the people who donated to the py3k proposal.

Here's an update on the recent progress:

Thank you to everyone who has provided initial feedback on the PyPy3 2.1 beta
1 release. We've gotten a number of bug reports, most of which have been
fixed.
As usual, we're continually keeping up with changes from the default
branch. Oftentimes these merges come at a cost (conflicts and or
reintegration of py3k changes) but occasionally we get goodies for free, such
as the recent JIT optimizations and incremental garbage collection.
We've been focusing on re-optimizing Python 2 int sized (machine sized)
integers:

We have a couple of known, notable speed regressions in the PyPy3 beta release
vs regular PyPy. The major one being with Python 2.x int sized (or machine
sized) integers.

Python 3 drops the distinction between int and long types. CPython 3.x
accomplishes this by removing the old int type entirely and renaming the long
type to int. Initially, we've done the same for PyPy3 for the sake of
simplicity and getting everything working.

However PyPy's JIT is capable of heavily optimizing these machine sized integer
operations, so this came with a regression in performance in this area.

We're now in the process of solving this. Part of this work also involves some
house cleaning on these numeric types which also benefits the default branch.

cheers,
Phil

PyPy3 2.1 beta 1

Philip Jenvey — Tue, 30 Jul 2013 21:35:00 GMT

We're pleased to announce the first beta of the upcoming 2.1 release of
PyPy3. This is the first release of PyPy which targets Python 3 (3.2.3)
compatibility.

We would like to thank all of the people who donated to the py3k proposal
for supporting the work that went into this and future releases.

You can download the PyPy3 2.1 beta 1 release here:

https://pypy.org/download.html#pypy3-2-1-beta-1

Highlights

The first release of PyPy3: support for Python 3, targetting CPython 3.2.3!
- There are some known issues including performance regressions (issues
  #1540 & #1541) slated to be resolved before the final release.

What is PyPy?

PyPy is a very compliant Python interpreter, almost a drop-in replacement for
CPython 2.7.3 or 3.2.3. It's fast due to its integrated tracing JIT compiler.

This release supports x86 machines running Linux 32/64, Mac OS X 64 or Windows
32. Also this release supports ARM machines running Linux 32bit - anything with
ARMv6 (like the Raspberry Pi) or ARMv7 (like Beagleboard,
Chromebook, Cubieboard, etc.) that supports VFPv3 should work.

Windows 64 work is still stalling and we would welcome a volunteer to handle
that.

How to use PyPy?

Cheers,
the PyPy team

Py3k status update #11

Philip Jenvey — Wed, 12 Jun 2013 19:17:00 GMT

This is the 11th status update about our work on the py3k branch, which we
can work on thanks to all of the people who donated to the py3k proposal.

Here's some highlights of the progress made since the previous update:

PyPy py3k now matches CPython 3's hash code for
int/float/complex/Decimal/Fraction
Various outstanding unicode identifier related issues were
resolved. E.g. test_importlib/pep263/ucn/unicode all now fully pass. Various
usage of identifiers (in particular type and module names) have been fixed to
handle non-ascii names -- mostly around display of reprs and exception
messages.
The unicodedata database has been upgraded to 6.0.0.
Windows support has greatly improved, though it could still use some more
help (but so does the default branch to a certain degree).
Probably the last of the parsing related bugs/features have been taken care
of.
Of course various other smaller miscellaneous fixes

This leaves the branch w/ only about 5 outstanding failures of the stdlib test
suite:

test_float
1 failing test about containment of floats in collections.
test_memoryview
Various failures: requires some bytes/str changes among other things (Manuel
Jacob's has some progress on this on the py3k-memoryview branch)
test_multiprocessing
1 or more tests deadlock on some platforms
test_sys and test_threading
2 failing tests for the New GIL's new API

Probably the biggest feature left to tackle is the New GIL.

We're now pretty close to pushing an initial release. We had planned for one
around PyCon, but having missed that we've put some more effort into the branch
to provide a more fully-fledged initial release.

Thanks to the following for their contributions: Manuel Jacob, Amaury Forgeot
d'Arc, Karl Ramm, Jason Chu and Christian Hudon.

cheers,
Phil

Py3k status update #10

Philip Jenvey — Tue, 05 Mar 2013 20:00:00 GMT

This is the tenth status update about our work on the py3k branch, which we
can work on thanks to all of the people who donated to the py3k proposal.

There's been significant progress since the last update: the linux x86-32
buildbot now passes 289 out of approximately 354 modules (with 39 skips) of
CPython's regression test suite.

That means there's only 26 test module failures left! The list of major items
remaining for 3.2 compatibility are now short enough to list here, with their
related tests:

Tokenizer support for non-ascii identifiers

test_importlib

test_pep263

memoryview (Manuel Jacob's tackling this on the py3k-memoryview branch)

test_memoryview

multiprocessing module currently deadlocks

test_multiprocessing

Buggy handling of the new extended unpacking syntax by the compiler:

test_unpack_ex

The new Global Interpreter Lock and new thread signal handling

test_threading

test_threadsignals

test_sys

Upgrade unicodedata to 6.0.0 (requires updates to the actual unicodedata
generation script)

test_ucn

test_unicode

test_unicodedata

CPyExt

test_capi (currently crashes)

Update int's hash code to match to CPython (float's is already updated on the
py3k-newhash branch. note that PyPy 2.x doesn't even totally match
CPython's hashing)

test_decimal

test_fractions

test_numeric_tower

Miscellaneous:

test_complex

test_float

test_peepholer

test_range

test_sqlite (a new cffi based version seems to be coming)

test_ssl

test_struct

test_subprocess

test_sys_settrace

test_time

Additionally there are still a number of failures in PyPy's internal test
suite. These tests are usually ran against untranslated versions of PyPy during
development. However we've now began running them against a fully translated
version of PyPy on the buildbot too (thanks to Amaury for setting this
up). This further ensures that our tests and implementation are sane.

We're getting closer to producing an initial alpha release. Before that happens
we'd like to see:

further test fixes
the results of test runs on other major platforms (e.g. linux x86-64 and osx
seem to have some additional failures as of now)
some basic real world testing

Finally I'd like to thank Manuel Jacob for his various contributions over the
past month, including fixing the array and ctypes modules among other things,
and also Amaury Forgeot d'Arc for his ongoing excellent contributions.

cheers,
Phil

Py3k status update #9

Philip Jenvey — Thu, 10 Jan 2013 06:04:00 GMT

This is the ninth status update about our work on the py3k branch, which
we can work on thanks to all of the people who donated to the py3k
proposal.

Just a very short update on December's work: we're now passing about 223 of
approximately 355 modules of CPython's regression test suite, up from passing
194 last month.

Some brief highlights:

More encoding related issues were addressed. e.g. now most if not all the
multibytecodec test modules pass.
Fixed some path handling issues (test_os, test_ntpath and
test_posixpath now pass)
We now pass test_class, test_descr and almost test_builtin (among
other things): these are notable as they are fairly extensive test suites of
core aspects of the langauge.
Amaury Forgeot d'Arc continued making progress on CPyExt (thanks again!)

cheers,
Phil

Py3k status update #8

Philip Jenvey — Tue, 04 Dec 2012 22:30:00 GMT

This is the eight status update about our work on the py3k branch, which
we can work on thanks to all of the people who donated to the py3k
proposal.

Just a short update on November's work: we're now passing about 194 of
approximately 355 modules of CPython's regression test suite, up from passing
160 last month. Many test modules only fail a small number of individual tests
now.

We'd like to thank Amaury Forgeot d'Arc for his contributions, in particular he
has made significant progress on updating CPyExt for Python 3 this month.

Some other highlights:

test_marshal now passes, and there's been significant progress on
pickling (thanks Kenny Levinsen and Amaury for implementing
int.{to,from}_bytes)
We now have a _posixsubprocess module
More encoding related fixes, which affects many failing tests
_sre was updated and now test_re almost passes
Exception behavior is almost complete per the Python 3 specs, what's mostly
missing now are the new __context__ and __traceback__ attributes (PEP
3134)
Fixed some crashes and deadlocks occurring during the regression tests
We merged the unicode-strategies branch both to default and to py3k: now we
have versions of lists, dictionaries and sets specialized for unicode
elements, as we already had for strings.
However, for string-specialized containers are still faster in some cases
because there are shortcuts which have not been implemented for unicode yet
(e.g., constructing a set of strings from a list of strings). The plan is to
completely kill the shortcuts and improve the JIT to produce the fast
version automatically for both the string and unicode versions, to have a
more maintainable codebase without sacrificing the speed. The autoreds
branch (already merged) was a first step in this direction.

cheers,
Philip&Antonio