<tos9>
fijal: obviously out of date with the discussion, but yeah maybe it is a start?
<fijal>
👍
dddddd has joined #pypy
<ronan>
mattip, tos9: I think the failing tests are wrong, because they check a condition that should never occur and are incompatible with zlib 1.2.8
<tos9>
ronan: the "inconsistent state" ones were meant to handle what zlib.h says happens when somehow stuff isn't fully initialized
<tos9>
ronan: which still seems to be there at least in 1.2.11: "Z_STREAM_ERROR if the source stream state was inconsistent (such as zalloc being Z_NULL)."
<tos9>
(I couldn't see how to get zlib itself to give me that invalid state, so I manufactured it myself)
<ronan>
that explains why it's only deflateCopy() that segfaults, and not inflateCopy()
<tos9>
ah, indeed -- in 1.2.11 the two look more similar
<kenaan>
arigo unicode-utf8 fb5c9143c946 /pypy/objspace/std/unicodeobject.py: Add a conditional_call_elidable here
xcm has quit [Read error: Connection reset by peer]
xcm has joined #pypy
mattip has quit [Ping timeout: 250 seconds]
gsnedders has quit [Quit: leaving]
gsnedders has joined #pypy
gsnedders has quit [Client Quit]
<Ninpo>
Anywhere I can grab the 7.0 cut/cheat and pass it to pyenv to run some tests? I'm mid dev on a project currently using 3.5 6.0 as it's orders of magnitude faster than cpython. If it'd help to have a 24h runtime on some heavy data I'd happily help
<Ninpo>
I've got lines that'd test the async comprehensions etc as well I can easily put back
<Ninpo>
also async generators
gsnedders has joined #pypy
kipras has joined #pypy
squeaky_pl has joined #pypy
_whitelogger has joined #pypy
<squeaky_pl>
are there any problems known problems on the current tag release, I'm seeing some zlib chat above?
<Ninpo>
ronan: ah it's the 3.6 I'm interested in for myself, but I'll happily test both if it's useful
<Ninpo>
Going to see if I can get pyenv to build the nightlies thanks for the links ronan
<ronan>
squeaky_pl: no, the zlib stuff is about a new feature that isn't in the release
<squeaky_pl>
Ninpo, why would you need pyenv? can't you just untar?
<kenaan>
fijal arm64 dba5b910fb11 /rpython/jit/backend/aarch64/: (arigo, fijal, rodolph, bivab) fight until the test nearly passes
<Ninpo>
squeaky_pl: tbh I was just thinking of the quickest way I could drop it into my virtualenv setup and switch easily
<Ninpo>
if it's prebuilt I can just drop it into versions
<squeaky_pl>
you can still untar, run new virtualenv on top of new pypy and install your program there
<squeaky_pl>
I updated all the dependencies and successfully translated 3.5 7.0 for portable builds on top of GCC 8.2, the resulting binary seems to work fine. I guess I am ready for the release.
<Ninpo>
squeaky_pl: I copied the pyenv installer helper for pypy3.5 nightlies and modified it :)
<squeaky_pl>
mattip_, are you gonna officially release on Monday?
<ronan>
squeaky_pl: Antonio is the release manager this time
<Ninpo>
:facepalm: pyenv's nightly installer is broken anyway.../me sighs, does it the "hard" way
<squeaky_pl>
ronan, thanks for the info
<fijal>
squeaky_pl: matti is in the plane right now I believe
<squeaky_pl>
I have enough RAM on my laptop to run 3 translations in parallel, how times changed.
<squeaky_pl>
Ninpo, those are built on Ubuntu, if your os version doesnt match exactly you might have problems.
<Ninpo>
oh I see.
<squeaky_pl>
Ninpo, are you trying to use 3.6 7.0?
<Ninpo>
yes
<Ninpo>
on Arch
<squeaky_pl>
Ninpo, I will have a portable build in more or less 1 hour, I will ping you.
<Ninpo>
squeaky_pl: oh wow thank you so much!
<squeaky_pl>
It will work or any x86_64 Linux that is 10 years or newer.
<Ninpo>
'tis why I was fiddling with pyenv and hoping to get it to work, but it falls over on 3.5 nightlies as well which have a install option provided
<Ninpo>
Appreciate the effort squeaky_pl and happy to help test
<squeaky_pl>
Ninpo, libtinfo comes from ncurses, you must have ncurses 6 installed on your distro or your distro doesnt provide libtinfo because the symbols are inside libncurses
<Ninpo>
ah probably
<Ninpo>
rolling release and all that
<Ninpo>
I take it the pypy3.5v6 is a "portable build"
<Ninpo>
that works fine with pyenv install
<squeaky_pl>
PyEnv can install my builds automatically, they package portable pypy
<Ninpo>
cor, 7 is going to be worth it for me whether it's 3.5 or 3.6. Very very nice work, thanks so much pypy folks
<Ninpo>
squeaky_pl: your repo was the missing piece of the puzzle, piece of cake creating pyenv install files now (and both versions installed)
<Ninpo>
Thank you very much. Off to test!
xcm has quit [Remote host closed the connection]
xcm has joined #pypy
themsay has joined #pypy
beystef has left #pypy [#pypy]
<Ninpo>
Overall runtime seems relatively the same which is nice. I've noticed processing of a db result set into a json structure is a bit slower, I'll test that locally on the DB in case it's network messing with it and provide a test case later if confirmed. Back later, thanks again squeaky_pl
<cfbolz>
Ninpo: thanks, if you find a slowdown please report it
<Ninpo>
cfbolz: I will, will take a little bit to scrub the data (lots of PII) but absolutely
<Ninpo>
This project would have been a non starter without pypy
squeaky_pl has quit [Ping timeout: 244 seconds]
<fijal>
Ninpo: can you share more with us about the project?
<fijal>
If you already did I'm sorry, on the phone
<Ninpo>
I haven't, at least not today :) I'm processing a bunch of text type fields in a mysql database looking for incorrectly encoded bytes (utf8 inside mysql latin1 fields) and saving the results to a json file per DB. Over 2TB of data
<Ninpo>
Using trio and sqlalchemy/sqlalchemy_aio for the heavy lifting
<Ninpo>
4 workers for db, 8 table workers per db worker
<Ninpo>
cpython managed 95k tables in about 10 hours last time I ran it (about 25% through), pypy35-6 does a complete scan in about 14 hours
<Ninpo>
There's a refactor run on the main logic coming up Monday now that a full successful pass has been done
<Ninpo>
Long running, all python, perfect for pypy
<fijal>
The slowest part is probably SQL alchemy
<fijal>
We never figured out how to speed up current orms
<Ninpo>
I'm not ORMing with it. I use the query builder in one spot, the rest is conn.execute
<Ninpo>
makes use of threads with sqlalchemy_aio
<Ninpo>
Part of the refactor will be to test trio_mysql which is a trio-ised pymysql
<fijal>
Yeah still
<fijal>
Try without it on a small data
<Ninpo>
'tis on the list :)
<fijal>
I'm happy to look into simple examples how to speed it up
<Ninpo>
One of the todos next week is come up with some varying size test data so the tests can be written, I'll make those available
<Ninpo>
bulk of the processing time is feeding from an async generator off the result set and binascii().decode operations
<Ninpo>
It was my intent to open source it, was on my personal github but my employer got upset :|
<Ninpo>
testing against individual schemas atm, I'll leave a run going overnight to see how the over time and memory performance is, I've got good 3.5v6 baselines for that