cfbolz changed the topic of #pypy to: PyPy, the flexible snake (IRC logs: https://botbot.me/freenode/pypy/ ) | use cffi for calling C | "the modern world where network packets and compiler optimizations are effectively hostile"
rokujyouhitoma has quit [Ping timeout: 248 seconds]
Tiberium_ has joined #pypy
Tiberium_ has quit [Client Quit]
Tiberium has quit []
Tiberium has joined #pypy
ronny has quit [Ping timeout: 240 seconds]
Tiberium has quit [Remote host closed the connection]
ronny has joined #pypy
Tiberium has joined #pypy
antocuni has quit [Ping timeout: 260 seconds]
rokujyouhitoma has joined #pypy
rokujyouhitoma has quit [Ping timeout: 260 seconds]
<exarkun>
good monday
<LarstiQ>
same to you exarkun
<exarkun>
fijal: So, what do you think? test_pyexpat? Something else?
yuyichao has joined #pypy
marr has quit [Remote host closed the connection]
nimaje has joined #pypy
rokujyouhitoma has joined #pypy
ronan has quit [Ping timeout: 260 seconds]
ronan has joined #pypy
rokujyouhitoma has quit [Ping timeout: 255 seconds]
<exarkun>
Okay well I looked at the test_pyexpat failure.
<exarkun>
It's pretty stupid. It asserts specific things about the traceback of exceptions from the parser (ie, what source files it references). The test should be skipped I think.
antocuni has joined #pypy
demonimin_ has quit [Ping timeout: 240 seconds]
demonimin has joined #pypy
<kenaan>
exarkun py3.5 e2a71b4f385a /: Replace fragile upstream expat exception test with a more reasonable one
<Cheery>
can the GIL-removal be implemented in rpython, or does it require lot of changes into the PyPy code as well?
<fijal>
Cheery: unfortunately it does
<Cheery>
is there a place for this where I can stay up-to-date on what's going on, and what needs to be done in my language's runtime to follow on?
<fijal>
Cheery: so what's gonna go now is we'll see if there is commercial interest associated with hype
<fijal>
if not, we can say "well all this talk, but noone actually cares"
<fijal>
for some definition of "noone"
<Cheery>
fijal: that's ok to me.
<exarkun>
fijal: I think "use utf-8" fixes test_readline. the state of pyrepl is pretty miserable, though. maybe someone implemented the wrong/complicated thing for some reason but who knows.
adamholm_ has quit [Remote host closed the connection]
adamholmberg has joined #pypy
adamholmberg has quit [Read error: Connection reset by peer]
adamholmberg has joined #pypy
rokujyouhitoma has joined #pypy
<simpson>
Hi! Congrats on the current nogil work. How much can be reused by non-PyPy RPython code? Should I just read the branch?
rokujyouhitoma has quit [Ping timeout: 255 seconds]
<arigato>
simpson: hi!
<arigato>
right now, the branch only fixes the GC
<arigato>
and turns off the GIL for RPython code
<arigato>
you can give it a try if your RPython code is thread-safe and you don't mind not having the JIT
<simpson>
arigato: Good morning! So, our RPython code is already structured with something like subinterpreters (we call them "vats") and we already take a per-vat thread lock as needed and use thread-local storage for vat locals. Basically we intended to allow multiple vats to run simultaneously via threading.
<simpson>
Ah, no JIT. Not the worst tradeoff.
<arigato>
if your RPython code is intended to be used without GIL protection, then yes, it's exactly our target
<simpson>
Swag. I can't devote a bunch of time to it Right Now, but this is a very important part of our overall strategy, so I'll hopefully start working on this over the next few weeks.
<simpson>
This is seriously cool work. What's next to get the JIT working? Does the JIT need a lock?
<arigato>
yes, we didn't think it through exactly, but one or a few locks should do it
<arigato>
but there are likely a few local problems
<arigato>
for example, code invalidation requires patching the assembler, which might now be executed by another thread
<simpson>
Oh jeez. This is like Cliff Click's nightmare slides, where he shows all the assembler around calls.
<simpson>
Sounds pretty tough.
<arigato>
we didn't think it through so far
<arigato>
there might be easish solutions involving switching the pages "not executable" for a moment, and catching any other thread trying to execute them with a signal handler
<arigato>
or else, we need to acquire a global lock and wait until all threads are stopped
<arigato>
which is needed anyway e.g. for major collections
<arigato>
or, we can ensure that patchable areas are within 16-bytes cache lines, which we can update atomically, but I'm not sure that's fully safe
<arigato>
or maybe the CMPXCHG128 instruction can be used for that purpose?
<arigato>
CMPXCHG16B
<arigato>
ah, it requires the 16 bytes to be fully aligned, too
<arigato>
simpson: maybe Monte can be a nice intermediate step to test and debug problems inside RPython
raynold has joined #pypy
bauerj has quit [Quit: No Ping reply in 180 seconds.]
gumblex has quit [Ping timeout: 240 seconds]
bauerj has joined #pypy
gumblex has joined #pypy
<simpson>
arigato: It's true that we designed around the eventual day when we had to write low-level threads in RPython or Rust or JS or whatever.
oberstet has quit [Ping timeout: 255 seconds]
kipras`away is now known as kipras
oberstet has joined #pypy
rokujyouhitoma has joined #pypy
rokujyouhitoma has quit [Ping timeout: 240 seconds]
<kenaan>
rlamy default 9ac0b00b959a /pypy/module/cpyext/: Do not leak reference when using PyModule_AddObject (reverts 5e783074ef82)
<LarstiQ>
hmm, is there a writeup somewhere about doing encodings/unicode in python by keeping it encoded as long as possible? Counter to the https://docs.python.org/3.6/howto/unicode.html advice of "Software should only work with Unicode strings internally, decoding the input data as soon as possible and encoding the output only at the end."
<exarkun>
I hope not
<kenaan>
rlamy multiphase 14b50017ab08 /pypy/module/cpyext/typeobject.py: Fix leaks in PyType_FromSpec
rokujyouhitoma has joined #pypy
rokujyouhitoma has quit [Ping timeout: 260 seconds]
<arigato>
ronan: 9ac0b00b959a: I can only guess you have some more information than I did when I wrote 5e783074ef82
<arigato>
which was written as a workaround for a very common problem in C extension modules
<arigato>
so if you blindly revert it, you're bringing this class of problems back
<LarstiQ>
exarkun: well, kinda important approach for pypy afaics
<exarkun>
LarstiQ: There's a difference between writing an application and writing a unicode abstraction, though.
<exarkun>
A bit one.
<exarkun>
s/bit/big/
<LarstiQ>
exarkun: there's also a difference between displaying ui text and manipulating data
<mjacob>
i'd guess that the fact that the "remove gil" blog post has more points on hacker news than "A Solution of the P versus NP Problem" shows that there is some interest in a GIL-less python ;)
<exarkun>
byte strings, for the most part, are not data you can manipulate.
<exarkun>
You can shovel them from here to there
<exarkun>
Or you can decode them
<mjacob>
(although the P != NP thing was posted one hour later)
<LarstiQ>
exarkun: sure, manipulate the containers, not the contents
<simpson>
mjacob: "I'd like to use PyPy, but I hear that the C/API doesn't work and that's a showstopper even though we don't have any C code, because Business Reasons." Saved you having to click~
<LarstiQ>
mjacob: people assume it's crackpot?
<exarkun>
LarstiQ: Sure. So make byte decoding lazy in the runtime. But don't _not_ decode it if it's passing through your program.
<mjacob>
LarstiQ: which one?
<LarstiQ>
mjacob: P != NP
<exarkun>
LarstiQ: Perhaps you aren't manipulating it today but chances are you will be later.
<LarstiQ>
exarkun: hrmf
<mjacob>
simpson: i'm trying to understand how the sentence is related to the GIL. is it a quote?
<LarstiQ>
exarkun: I guess the details aren't articulated well enough
<simpson>
mjacob: Oh, I'm just channeling the typical HN comment on any PyPy article~
<mjacob>
simpson: i'm not a regular HN reader, but my usual approach is "skim through the comments for good comments and ignore the bad ones"
<simpson>
mjacob: The best commenter is this person "fijal" who I don't think I know~
<mjacob>
simpson: my "favorite" comment so far: "Super glad they're going to try using mutexes and not that STM approach which was looking to be immensely complicated. Was not looking forward to the kinds of interpreter bugs that was going to produce."
<mjacob>
simpson: this guy probably thinks that the GIL is only there because nobody tried to remove it
<simpson>
Ha! Yeah.
<simpson>
It's interesting that people don't realize that GHC Haskell has been doing production-ready STM for years. The basic concept of STM has already been well-validated. It's just not easy.
exarkun has quit [Read error: Connection reset by peer]
<mjacob>
(my last message was a clear contradiction to "skim through the comments for good comments and ignore the bad ones" is the message before ;))
<mjacob>
yes, but also STM is quite different is a purely functional environment
<mjacob>
s/is/in/
<simpson>
I'm happy to see that there's a big subthread talking about Python, but with a raft of deep improvements to the language. That's validating the Julia/Monte approach IMO.
<simpson>
Not that Python is dying, of course. Python is the new Java. Yay.
<mjacob>
purely functional programming is the future
<mjacob>
(i hope)
<mjacob>
anyway, we are going far too off-topic for this channel
exarkun has joined #pypy
<mjacob>
arigato: do you think that the approach of marking some external c functions as "safe to call in replay mode" is possible in principle?
<mjacob>
arigato: i know that there are a few things that need to be solved before, but i'm happy to help with these
<arigato>
mjacob: maybe, but keep in mind that this function cannot receive any pointer
<arigato>
in replay mode, any raw pointer is guaranteed to have the same value as it had during recording, and no actual memory behind it
<mjacob>
arigato: well, if i stack allocate some variable, pass it to the function, read it once afterwards, it shouldn't be a problem
<arigato>
in RPython? we have no stack allocation
<mjacob>
s/stack allocate/allocate and free afterwards/
<arigato>
ok, then no, that's wrong
<arigato>
allocation returns the same address as it returned during recording, and doesn't really allocate anything
<arigato>
you can do that in C, of course
<arigato>
but not in RPython
<mjacob>
so currently e.g. in strtod, what's happening with the address of the temporarily allocated memory so that the c function can "return" the end pointer?
<mjacob>
that sentence it wrong grammtically, but i mean the "end_ptr" variable in strtod
<arigato>
hum, ok, rdtoa.strtod() is a more complicated story
<mjacob>
(asked differently, how is mallocing and freeing something temporarily problematic?)
<arigato>
strtod() does two different things depending on revdb.flag_io_disabled()
<arigato>
I forgot that point
<arigato>
the common case is that flag_io_disabled() is false
<mjacob>
arigato: well, my goal is to remove that hack ;)
<arigato>
well, good luck
<arigato>
I don't think it's easy
Rhy0lite has quit [Quit: Leaving]
<arigato>
q
<arigato>
(sorry)
rokujyouhitoma has joined #pypy
<mattip>
arigato: about 9ac0b00b959a, do you happen to remember what c extension (ab)used those functions?
<ronan>
arigato: the problem is that the hack did cause problems for extensions using PyModule_AddObject correctly
<mattip>
(hi ronan)
<arigato>
ronan: ok
rokujyouhitoma has quit [Ping timeout: 248 seconds]
<arigato>
mattip: no, sorry
<arigato>
several ones, though
<arigato>
(including some I threw together myself long ago)