antocuni changed the topic of #pypy to: PyPy, the flexible snake (IRC logs: https://botbot.me/freenode/pypy/ ) | use cffi for calling C | "PyPy: the Gradual Reduction of Magic (tm)"
danieljabailey has quit [Quit: ZNC 1.6.5+deb2build2 - http://znc.in]
danieljabailey has joined #pypy
pulkitg has quit [Read error: Connection reset by peer]
yuvipanda has quit [Remote host closed the connection]
abvi[m] has quit [Write error: Connection reset by peer]
bendlas has quit [Remote host closed the connection]
Goto has joined #pypy
pulkitg has joined #pypy
danchr has quit [Read error: Connection reset by peer]
<koroep>
Hi there! Just tried to run pypy on my win10. No problems until I try to install pip with pypy3.exe -m ensurepip. v5.10.0-win32 gives me: "AttributeError: module 'errno' has no attribute 'EWOULDBLOCK'" and the latest nightly: "NameError: name 'long' is not defined". Is this a bug or am I doing something wrong here? :)
Rhy0lite has joined #pypy
adamholmberg has joined #pypy
<arigato>
cfbolz: right
<arigato>
cfbolz: it says "This is a fully public sprint: newcomers and topics other than
<arigato>
those proposed below are welcome."
ionelmc has joined #pypy
raynold has quit [Quit: Connection closed for inactivity]
tayfun26 has quit [Remote host closed the connection]
antocuni has quit [Read error: Connection reset by peer]
pilne has quit [Quit: Quitting!]
antocuni has joined #pypy
marr has quit [Ping timeout: 268 seconds]
aboudreault has joined #pypy
* mattip
back
<mattip>
arigato: about issue 2730, changing the file name to wow.txt still succeeds in cpython3, fails on py3.5
<mattip>
is \w also suspicous? I did make sure "dir abcαßΓ" shows the file
jcea has joined #pypy
<arigato>
just saying, I don't trust Windows parsing inside single- and double-quotes
<arigato>
of the command line
<arigato>
I guess you get the same results by typing the commands interactively, but could you check?
<mattip>
yup, same failure from the repl
<mattip>
when I try ctypes.CDLL(r'abcαßΓ\wow.txt'), on CPython it says 'invalid DLL', on PyPy3 I get a UnicodeEncodeError: 'mbcs' codec can't encode ...
<arigato>
ok, and you know where it comes from?
<arigato>
or should I debug on Windows
<mattip>
is it a blocker for 5.10.1?
<mattip>
I can debug it
<mattip>
but maybe we should do 5.10.1 first?
<arigato>
I would say it is not a blocker, otherwise we'll have a long chain of blockers for Windows
<arigato>
let's making 5.10.1 that at least works a little bit more
<arigato>
probably worth writing in the release notes that there are still at least one and probably many issues like that on windows
<arigato>
(and thus it's beta)
<arigato>
(and please report etc.)
<mattip>
+1, release after merging py3-winreg
<arigato>
cool
<mattip>
hmm. I only want the winreg branch + fixes, not the merge from default in 799e8d00a8ec
<mattip>
I guess I have to graft each commit
<mattip>
I guess the release notice should also say something like "get your macos PyPy from homebrew, unless you are a developer"
<arigato>
likely
<fijal>
we should fix it one day
<fijal>
but if it means "fijal fixes it one day", that might need to wait
<arigato>
mahmudov: "how can I pass out of memory issue" => that's what fijal just said above
<arigato>
in more details, if you want a safer explicit way, start "rpython" with the "--source" option, then go to /tmp/xxx/testing_1 as printed as the end, and type "make"
<mahmudov>
hm after "make" it will finish compiling?
<fijal>
you'll get a pypy-c
<mahmudov>
then?
<fijal>
you would still need to copy it by hand somewhere and run the build_cffi_modules step
<fijal>
but copying pypy-c and libpypy-c.so from that directory should be enough
nunatak has quit [Quit: Leaving]
<arigato>
the build_cffi_module step is, I think, running pypy/tool/build_cffi_modules.py with the newly built pypy-c
<fijal>
and then ./pypy-c pypy/tool/build_cffi_imports.py I think
<fijal>
(in the pypy checkout directory where you just copied pypy-c from)
<fijal>
that gives you a fully working pypy-c in a directory that you can move somewhere
<mahmudov>
hm also a weird thing when it is killed
<mahmudov>
but cc is still running
<mahmudov>
but console output is stopped
<fijal>
if you had swap, it would not be killed
<arigato>
likely an out-of-memory, yes
<fijal>
(since you start an extra C compiler on top)
<arigato>
random things get killed, but the translation itself is a big process so likely to be the target
tbodt has joined #pypy
<mahmudov>
cant we make it slower ?
<mahmudov>
or reduce memory usage
<arigato>
follow the steps we just describes reduces memory requirements a lot
drolando has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
drolando has joined #pypy
tos9 has quit [Ping timeout: 264 seconds]
<ronan>
arigato: bencher4 is up though, but the buildslave is not
<mahmudov>
whats the diffrence translating with JIT or without?
drolando has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
Taggnostr2 has quit [Ping timeout: 264 seconds]
drolando has joined #pypy
drolando_ has joined #pypy
drolando has quit [Ping timeout: 265 seconds]
jamesaxl has quit [Quit: WeeChat 1.9.1]
mahmudov has joined #pypy
* arigato
playing with mmap-for-arenas again
<arigato>
I can't seem to understand something even on standard pypy
<arigato>
at once point I have 2000 arenas in the program, plus a small extra fraction of raw-malloc
<arigato>
but the program takes 2.6GB according to the kernel
<arigato>
2000 arenas take 1GB
<njs>
any hints in /proc/$PID/maps, maybe?
<arigato>
no, I see mainly a single 2.6GB region
<arigato>
very severe fragmentation?
<xorAxAx>
if they have the same size, why would there be fragmentation?
<arigato>
I know there is a little bit of fragmentation, coming from the fact that they are the same size but there is also the 10% extra of various sizes
<arigato>
all bunched together in the same malloc pool
<arigato>
but that's probably not it
<xorAxAx>
so the branch does a single mmap for all arenas?
<arigato>
yes, which should avoid this little bit of fragmentation
<arigato>
(which may be important in some cases, but likely not in translate.py)
<arigato>
(also, I'm fighting this in the context that the mmap-for-arenas branch uses *more* memory than default, which does not make sense either)
<xorAxAx>
and the mapping is 2.6 GB or the used space?
<arigato>
the mapping is 2.6 GB but all is in RSS
<arigato>
private pages etc.
<xorAxAx>
which makes it sound likely that mmap() was called with that size
<arigato>
I guess it's malloc() that called mmap() many times, and the kernel merges adjacent memory regions with the same flags
<xorAxAx>
ah, in that case, it will likely do some rounding, leading to fragmentation, indeed
<xorAxAx>
does the GC have a concept of deallocating an arena with /dev/zero copies to it?
<xorAxAx>
in case its not needed anymore
<xorAxAx>
or would it munmap
<arigato>
on default, it uses madvice(MADV_FREE)
<xorAxAx>
and on the branch?
<arigato>
it munmap, indeed
<arigato>
which looks like a good idea because the major GCs are rare enough and the total number of pages relatively constant
dddddd has quit [Remote host closed the connection]
<arigato>
...total number of arenas
<xorAxAx>
except if the kernel logic doesnt support splitting mappings
<arigato>
it does
<xorAxAx>
can you increase the arena size to see if the 2.6-2 difference changes?
<xorAxAx>
(or lower the arena size)
<arigato>
why 2.6-2?
<arigato>
the difference is between 1GB and 2.6GB
<arigato>
yes, I could, but that would take one hour of computations
<arigato>
I'm unsure to understand why
<koroep>
arigato: Thank you for helping me earlier!
<koroep>
See you all
<arigato>
see you!
koroep has quit [Quit: Page closed]
<njs>
arigato: can you get the GC to make a map of which addresses are actually in use, maybe draw a picture?
<xorAxAx>
arigato: ah, maybe there is some obvious pattern if you vary the arena sizes
<xorAxAx>
are you setting MALLOC_MMAP_MAX_?
<arigato>
no, so the mallocs are below the limit
<xorAxAx>
i.e. there are less than 2**16 malloc() calls with more than x kB of size? ok
<arigato>
why 2**16?
<xorAxAx>
that is the default according to a website i found
<arigato>
ok, then more precisely, I have little clue about the details of malloc
<arigato>
I'm not setting anything
<xorAxAx>
try MALLOC_MMAP_MAX_=200000 ./binary
<arigato>
yes, I know its approximative effect
<xorAxAx>
so it doesnt become better?
<arigato>
the situation is summarized in some issue, I can't find it any more
<arigato>
ok what. now I'm running again, I see almost the same number of arenas and the same memory used inside the arenas, but now the system reports 1.7GB
<arigato>
...which just jumped to 2.0GB even though the numbers didn't change
john51_ has quit [Ping timeout: 246 seconds]
<arigato>
...no, just confused I guess, it's now at 2.3GB and the numbers didn't change
<arigato>
maybe it was the same previously
<arigato>
it still makes no sense of course, but at least it's the same as the previous time
<xorAxAx>
yeah
john51 has joined #pypy
exarkun has quit [Ping timeout: 256 seconds]
exarkun has joined #pypy
adamholmberg has quit [Remote host closed the connection]
adamholmberg has joined #pypy
pilne has joined #pypy
adamholmberg has quit [Remote host closed the connection]