cfbolz changed the topic of #pypy to: PyPy, the flexible snake (IRC logs: https://botbot.me/freenode/pypy/ ) | use cffi for calling C | the secret reason for us trying to get PyPy users: to test the JIT well enough that we're somewhat confident about it
<kristina>
also i passed --no-shared. and it didn't do --no-shared because it links against things that it should not link against.
<kristina>
so i take it, i need to recompile /DEPOT/pypy3-v6.0.0/usession-release-pypy3.5-v6.0.0-0/testing_1 but now i need to actually pass in proper flags instead of random stuff pypy adds which is completely bogus?
<kristina>
it appears that the build system is really upset because it clobbers the checkout directory. that's very illegal :(
<kristina>
it should not emit any artifact that traverses outside /DEPOT/pypy3-v6.0.0/.
wallet42 has quit [Ping timeout: 240 seconds]
<LarstiQ>
kristina: --no-shared at least means to not build _libpypy.so_ as a shared object separate from pypy-c, I'm not sure it means statically linking all dependencies
<kristina>
oh so i want --shared and then to decompose it to something that can be statically linked?
<kristina>
that did the opposite to what i wanted i guess.
<LarstiQ>
kristina: I understand your frustration, but "completely bogus" and "very illegal" may not be the most help-inducing formulations
<kristina>
well it's illegal to put things in the non artifact directory unless you're installing in which case it goes into the virtual sysroot.
<LarstiQ>
kristina: that's a bit a matter of opinion
<LarstiQ>
and it's not, I think, that the pypy team is opposed to working on the build system to make it work like that, but neither is it a priority
<LarstiQ>
I'm sure patches are welcome, freebsd build is maintained by the community too
<kristina>
but it's not, i can ask clang to forcefully redirect output using its VFS schenanigans, but you have your own code generator too, so i have to ask the build system to inject something to deal with things that are unable to handle paths properly (it's either that or get warnings about clobbering the checkout directory or fail instantly).
<LarstiQ>
kristina: I'm sorry to be the bearer of bad news, but your environment is so far out the norm that it will be up to you to do the work
<LarstiQ>
unless you convince someone else
<kristina>
well it's normal to have an artifact directory.
<LarstiQ>
kristina: maybe have a look at how say Debian solved that?
<kristina>
the --no-shared is wrong, i misunderstood the flag.
<cfbolz>
kristina: some of the options seem redundant, eg --thread --continuation
<cfbolz>
Those should be on by default
<cfbolz>
Same for all the withmod options
<kristina>
what about the jit? also i still don't understand, is pypy a py->c compiler and/or a jit for stuff that's not precompiled or is it one of those?
<kristina>
as far as i understand it has an optional JIT to do that that you enable with --opt=jit and that other jit flag.
<fake-name>
AIUI, it JITs the bytecode, without a internal C step
<fake-name>
and from earlier, the JIT is on unless you specifically turn it off
<kristina>
what's --opt=jit?
<fake-name>
¯\_(ツ)_/¯
<fake-name>
is that a compile flag, or a runtime flag?
<cfbolz>
fake-name: they aren't really *build* failures. But some tests are failing right now
<fake-name>
well, the point stands that someone has automated builds
<kristina>
but currently it's 1). clobbering checkout (illegal) 2). [RIGIN/../lib:/usr/local/lib] though i shouldn't need that since i can roll it into a static object if it's built as a shared library (run a semi-static linker over it but that makes it ineligible for IPO). 3). so -Ojit is what i want to produce a pypy.so that JITs objects that weren't compiled by rpython? any flags i need aside from that?
<kristina>
and output would be $INSTALL_PREFIX/lib/pypy_stage1.so i guess since that's the "install" output path?
<fake-name>
the troll in me really, REALLY wants to point out s/illegal/I don't agree with this/, but other then that, I have no useful knowledge
<kristina>
you can do it locally, you can't commit code to the central repository that does things like that. there's a handy helper for it that injects itself into everything starting from the build system invocation (in that case shell script and hooks fs related system calls to do stuff which is either redirect/warn about misbehaving projects/fail on misbehaving projects).
<fake-name>
yeah, but that's still not 'illegal', just annoying
<cfbolz>
kristina: I may have missed this, but which OS/distribution are you doing this for?
<kristina>
yeah but for central repository it must not do it, a single warning about traversing outside it's buildroot is enough.
<fake-name>
yeah, but it's not *illegal*. Illegal implies there are laws about it
<kristina>
oh this is on linux + god knows what for userland. well there are laws of the central repository like thou shall not clobber the checkout.
<fake-name>
there are *standards* and hygiene, and lots of other things, but no laws
<fake-name>
but arguing about that is bikeshedding
<kristina>
it breaks buildbots, that's the reason it's not allowed.
<fake-name>
it's self evidently allowed, since it's already in-place
<kristina>
no it's on my computer.
<cfbolz>
kristina: sorry, I am still missing context. What is it that you are actually trying to do?
<fake-name>
from the context, set up a automated build system for an os?
<cfbolz>
Yes, but which distribution? Several Linux distributions build pypy just fine
<cfbolz>
kristina: hm, a thought: it's possible that running rpython with --source would be a way to tackle the problem. It should stop after generating the C source in the usession dir, so afterwards you would have to run make their, and build the cffi modules yourself
<kristina>
hm, i'll say think kinda libsystem.so that has most system libraries and their exports and has most of the dynamic linker (ie. it resolves against itself). some sort of amalgamation non-gnu linux.
<fake-name>
is this some weird embedded thing?
<kristina>
it's for aarch64 and x86_64 pretty much, i mean i wouldn't say embedded since most of them are proper higher end-ish application processors.
<fake-name>
point taken
<cfbolz>
kristina: does it have a name?
<kristina>
i mean it makes sense to use a single library since -latomic -ldl -lrt -pthreads (lsystemd if you're using systemd) etc, is going to be loaded in every process, you can run a single IPO pass over the entire thing.
<cfbolz>
Anyway, my suggestion would be to look into --source
<kristina>
yeah i was trying to but didn't find it anywhere in the docs.
<kristina>
it sounds like a i need a 2 pass build? one pass to get the files to compile and 2nd pass to invoke whatever you build with?
<cfbolz>
Something like that, yes. First run rpython --source -Ojit, that will give you the C source
<cfbolz>
Then run make in usession
<cfbolz>
Then build the cffi modules
<kristina>
so the --source will not ever clobber the checkout?
<cfbolz>
I am not promising anything ;-). That's really not a concern we have in mind (as you are discovering). But I think you have a decent chance that it doesn't
<kristina>
i mean a lot of things like clobbering the build root which is when you just add a cmakelists.txt/build.gn for them (depending on your mood) that handles it and scrap the original build system since they do the right thing. but rpython is not that simple i guess.
jsza has joined #pypy
<kristina>
cmake/gn make very few assumptions about the target that can't be overriden.
<kristina>
Makefiles on the other hand well, depends on how they're written.
<fake-name>
obviously the solution is to just use docker /s
<kristina>
if they respect env vars 100% makefiles just work (tm).
<kristina>
and i like ninjas more.
<kristina>
some more complex makefile based build systems are basically fork bombs.
<kristina>
which is fine but you kind of wonder why they went with this approach.
<fake-name>
I can pretty much gaurantee that 99% of the build system choices out there come down to "the guy who was setting that stuff up was most familliar with/really liked <thing>"
<kristina>
yeah but generally i never had issues with cmake or gn, aside from setting a few options, take rocksdb for example: http://crna.cc/VRoAX0mPtOlp4Hn
<kristina>
i didn't have to edit CMakeLists.txt or use a shell script to preconfigure it.
<kristina>
cmake 3.10 got OBJECT targets which is a godsend.
<cfbolz>
rpython is really its own thing. More of a compiler than a build system per se
<kristina>
also pretty colors break it, i'm not sure if it's a llvm bug with "is this a pty that supports escape sequences" detection or a rpython bug.
glyph has quit [Quit: End of line.]
glyph has joined #pypy
<kristina>
it has like a configure stage thing where it builds those little test thingies i guess to work out what the compiler can and can't do.
<kristina>
if lld or clang show a diagnostic, the escape sequences make rpython exit with something like unknown argument -<garbled character>.
<cfbolz>
kristina: uh, weird
<kristina>
this is either hitting a really odd corner case in llvmsupport or it's a bug in rpython.
glyph has quit [Client Quit]
<cfbolz>
kristina: any simple way I can reproduce?
glyph has joined #pypy
fake-name has quit [Ping timeout: 244 seconds]
glyph_ has joined #pypy
lritter has joined #pypy
glyph has quit [Ping timeout: 245 seconds]
glyph_ is now known as glyph
<kristina>
just retrying it myself without disabling colors ... seems to not happen anymore, i think that may've been a bug i fixed where it was overriding -fcolor-diagnostics in the clang's driver but it was also passed as a -Wl,--color-diagnostics (and i got rid of that too).
<kristina>
so when lld was invoked, unstripped -Wl,--color-diagnostics was passed to it.
<kristina>
and lld invoked by the driver doesn't do the check because, well, i guess that's also a bug but then again i doubt anyone calls ld.lld directly.
<kristina>
true question is, should the driver obey the -Wl and pass the flag down or not.
<kristina>
semantically it seems that -Wl, is "i really know what i'm doing just pass this down as is" since most common and cross-platform things you wouldn't need to pass explicit flags to whichever linker is invoked.
<kristina>
yes i guess this is not rpython's fault, the sequences should have been stripped, and i guess build system is to blame for using that flag instead of just letting the driver handle it and strip it before it called lld.
glyph has quit [Quit: End of line.]
glyph has joined #pypy
<kristina>
but yeah, in terms of the weird "libc", like the issue with musl is that for starters it's extremely poorly optimized, generally platform wise, i think more exotic arches like sh, or microblaze or m68k should maybe ship as an extra instead of mainline. glibc is even worse. musl is like really bizzare.
<kristina>
on linux it intentionally clobbers some syscalls with -ENOSYS.
<kristina>
some wrappers*
glyph_ has joined #pypy
<kristina>
i don't know why implement the wrapper only to break it, intentionally (because musl author really thinks that it's posix or nothing).
<kristina>
while oddly enough others do have wrappers.
glyph has quit [Ping timeout: 264 seconds]
<kristina>
also musl author shot down the proposal for dlopen from memory.
<kristina>
it's a very painful thing to do because you have to either locate and call linker's internal machinery (have to initialize TLV and register the object with the shared linker image within that task). just linking and relocating isn't enough.
agates has quit [Ping timeout: 240 seconds]
<kristina>
another big issue with glibc is that for some odd reason they decided that pthread_cancel was worth force-enabling exceptions and unwind tables.