sb0 changed the topic of #m-labs to: ARTIQ, Migen, MiSoC, Mixxeo & other M-Labs projects :: fka #milkymist :: Logs http://irclog.whitequark.org/m-labs
<whitequark>
since we'll have to deal with this numpy shit and so on
<whitequark>
and your argument that it does not have enough prebuilt packages applies.
<rjo>
yes. conda is a (albeit feeble) attempt to tackle it. unfortunately conda is not pain-free (even pretty painful). but again, thinking everything through to the scope that we need, if there remains an alternative it might be worth it.
<whitequark>
so... it seems we're stuck with conda forever? this is definitely not good news
<rjo>
no. others feel the same pain.
<whitequark>
I've already combed through binary package managers that work on both linux and windows
<whitequark>
there are like two and they don't work for our use case.
<rjo>
imho there is pressure on conda to get better.
<whitequark>
0install and yypkg
<whitequark>
conda is a massive trash fire
<whitequark>
we're going to see a proton decay before it gets better
<whitequark>
it has so many bugs that I stopped bothering to *report* them, do you think by this point it is likely that anyone will emerge with the capability to fix it ?
<whitequark>
it has no real governance, funding, architecture, or talent behind it
<rjo>
well. imvvho every package that we might want to require but that is not yet packaged well in conda should see more deliberation.
<whitequark>
the problem is not small packages, like lit or whatever, those work fairly well
<whitequark>
the problem is ... what the conda is supposed to solve, complex, large packages
<whitequark>
llvm, qt, numpy
<rjo>
yes. qt5 is obviously a challenge.
<whitequark>
qt was actually not that bad, i got it with just two builds
<whitequark>
the original packager just did many very questionable decisions
<whitequark>
the problem is not making any simple package for most part, it's that you are constantly bombarded with completely irrelevant bugs
<rjo>
but still there will be problems with qt because it (afaict) it may well conflict with e.g. ipython-qtconsole
<whitequark>
likely it does
<whitequark>
I'll even go further
<whitequark>
it is 100% certain it will conflict with all packages depending on qt on anaconda.org main repo right now
<whitequark>
since that doesn't have a qt5.
<whitequark>
and the APIs are not compatible.
<rjo>
so in the end it may have been less trouble to stick with qt4 and its problems/workarounds/docking-limitations/no-embedding.
<whitequark>
probably not
<whitequark>
that seems to me to significantly cripple GUI
<whitequark>
those NIST people use large amount of monitors, right?
<whitequark>
and embedding is super important for crashy widgets, and of course they will be crashy, written with shitty bindings in a shitty dynlang
<rjo>
two or three. but still. i don't think we have seen the story yet.
<rjo>
let's look at what people will use as an editor for the experiments. likely to be spyder. needs qt. will break. no editor = no fun.
* whitequark
laughs
<whitequark>
so do you know where I took the qt package? it was spyder's anaconda.org channel
<rjo>
qt5?
<whitequark>
yep
<rjo>
so spyder needs qt5?
<whitequark>
yep
<whitequark>
but their qt5 package is incompatible with ours, because they attempted to make it coexistable with qt4 and failed miserably
<whitequark>
if my understanding of qt.conf is correct, that is not possible at all with conda's layout
<whitequark>
and in any case that bug you encountered (with the platform plugin) was a direct consequence of spyder's sloppy packaging
<rjo>
will spyder-qt5 coexist with our qt (5 but named qt)?
<whitequark>
probably only on linux, since on windows our qt.conf's differ and spyder-qt5 will install its own over ours
<whitequark>
we, uh, can rebuild spyder to use our qt.
<rjo>
and we would be back at spyder conflicting with artiq, right?
<whitequark>
yes, if we do not rebuild it
<rjo>
(unless we build spyder)
<rjo>
ack
<whitequark>
for bonus points, it will probably break artiq silently
<whitequark>
since conda doesn't actually check for overlapping files in packages as far as I'm aware
<whitequark>
despite the fact that it has everything it needs, including precise installed file lists...
<rjo>
i have seen that as well.
* rjo
dreams of dpkg
<whitequark>
yes, dpkg would be a massive improvement.
<rjo>
if windows had it.
<whitequark>
even though building debian packages is a PITA in itself too
<whitequark>
at least when it breaks, it means you screwed up
<whitequark>
when conda breaks, it usually means a bug in conda.
<whitequark>
did I tell you how I accidentally put inconsistent data into anaconda.org and different parts of it thought the package had different version?
<rjo>
i have built debian packages for ~7 years. yes. it is hard but rewarding.
<whitequark>
building debian packages is /needlessly/ hard. something like a nix derivation is a much nicer way than a bizarre mix of makefile, shell and perl that's essentially undocumented
<rjo>
then summarizing: we are pretty certain that there will be a bunch of problems coming revolving around qt5/qt
<whitequark>
i am 100% sure there will be
<whitequark>
we can try fixing spyder's broken qt5, but then they could decide to upload a newer version
<whitequark>
which will break ARTIQ again
<rjo>
how can we preempt that in the most pragmatic way that is at least a bit future-proof?
<whitequark>
two conda channels with the same package cannot reasonably coexist
<whitequark>
so as long as we build our own qt, there will be problems of this sort.
<rjo>
we should tell them how to do it. maybe they will listen.
<whitequark>
oh, sure, but that won't solve the problem
<whitequark>
what if tomorrow spyder decides to package qt 5.7 and it breaks ARTIQ somehow?
<whitequark>
or, more likely
sb0 has quit [Ping timeout: 250 seconds]
<rjo>
how do qt4 and qt5 coexist on debian?
<whitequark>
what if tomorrow sb0 decides we need to ship qt with a patch and then spyder decides to package 5.7, which overrides our qt install?
<whitequark>
sb0 does in fact want to do such a thing.
<whitequark>
rjo: the problem with coexistence is that qt4 and qt5 in *portable* installs, like conda's, require a qt-version-specific qt.conf that is placed in the exact same location (next to python.exe)
<whitequark>
since qt's code for locating qt.conf was not really supposed to work on scripting languages
<whitequark>
it looks up qt.conf relative to the executable, which means every executable is free to ship its own
<whitequark>
if that worked, our problem would be largely fixed (we could put a qt.conf in artiq/frontend)
<rjo>
can't it be patched to look for qt5.6.conf?
<whitequark>
but that doesn't work because qt has no idea it's actually artiq_gui.py that's the executable
<whitequark>
it can
<rjo>
next to python3.5.exe
<whitequark>
no, conda ships python.exe
<whitequark>
python3.5 is a link
<rjo>
yes.
<whitequark>
and on windows, python3.5 does't appear in argv[0] as far as I know
<whitequark>
anyway
<rjo>
but would patching it to look for qt56.conf fix it?
<whitequark>
we can probably do a less invasive solution, which is set QT_CONF_PATH inside artiq_gui
<whitequark>
before loading pyqt
<whitequark>
but then we need to patch qt again into something like qt-artiq
<whitequark>
rjo: forget that
<whitequark>
we're arguing about bullshit
<whitequark>
why can't people install spyder into a separate environment?
<rjo>
maybe the spyder people are onboard once they realize that their qt5 is unworkable?
<whitequark>
rjo: that's not the problem here
<whitequark>
unless we *perfectly synchronize* our release cycles with spyder, we'll have mismatches
<whitequark>
that will break either of our packages
<whitequark>
if we ever decide to ship patched qt
<rjo>
that may be a viable but ugly workaround for spyder but it won't be for other qt-users.
<whitequark>
if we decide to never ever ship patched qt, then we can just instruct people to add spyder's channel
<whitequark>
that will work
<whitequark>
but shipping our own version of qt AND permitting users to add conda channels with other versions of qt is unworkable
<whitequark>
it may sometimes work by accident but it is in no way a working long-term solution since it will definitely break
<rjo>
you can expect people to do conda install anaconda which will pull in spyder (somehow) and do conda install artiq in the same env.
<whitequark>
so the one in anaconda would be the one pulled in from that metapackage
<whitequark>
it will break ARTIQ
<whitequark>
since there will be a conflict over versions of qt.
<rjo>
long story short: we need to coexist with a bunch of other packages in the same python env. i don't see a way around that.
<whitequark>
ok, that means i have to completely redo our qt packaging
<rjo>
at least artiq does not conflict with numba over llvmlite.
<whitequark>
and it will be slower this time
<whitequark>
though hopefully not *much* slower
<rjo>
well. maybe i don't see the full picture here and there is a way around it. but even applets are not a solution because of the embedding stuff right?
<whitequark>
what do you mean?
<whitequark>
how would applets help?
<whitequark>
oh, I see, embedding across Qt versions
<rjo>
can we embed a qt4 matplotlib applet into artiq qt5?
<rjo>
yes
<whitequark>
in theory, yes
<rjo>
sure. it is X11
<rjo>
;)
<whitequark>
ah yeah, that doesn't work on windows
<rjo>
and anyway. there is spyder and ipython-qtconsole, wx-qt, etc
<rjo>
by the way: i do have a lot of respect for your qt5 packaging
<whitequark>
I do not :)
<whitequark>
it's a sloppy job that I'm wholly unsatisfied with
<whitequark>
but, it's hard to improve it much without spending a lot more time on it...
<whitequark>
there should be more careful consideration for configure flags, buildscript that doesn't try to use two different MSVC versions at the same time
<whitequark>
I have no clue why that seems to work
<rjo>
ack
<whitequark>
I basically took spyder's package and did minimal fixing of obvious bugs in it to make it work with ARTIQ
<whitequark>
since sebastien wants everything faster, that seemed appropriate
<rjo>
i looked at nix for two minutes. isn't the fact that e.g. python (noarch) packages won't include "the hash part of Nix store paths" yet still depend on versions of other packages, a bit of a problem?
<whitequark>
I know that Nix packages python successfully, so they must have figured it out
<whitequark>
I think how it works is they have a place they symlink actual packages into
<whitequark>
and then it's just up to nix to crete a consistent view in that place
<rjo>
also IMHO one of the significant contributions of a distribution is that it manages ABI versions for you to the extent possible. the ultimately fine-grained stuff of nix seems guaranteed to bloat stuff.
<whitequark>
yes.
<whitequark>
this is why I don't use nix myself, and what nix fanboys ridicule me for.
<whitequark>
while nix is not a definite improvement over dpkg (transactionality is still sweet), it is a definite improvement over conda
<whitequark>
even if it bloats stuff.
<rjo>
if i fix a bug in libc i have to recompile and reinstall my entire system to undo the (factor two) bloat, right?
<whitequark>
yes
<whitequark>
well, there are binary packages
<whitequark>
but if you, specifically, fix it, then yes
<rjo>
Then I see now where all the gentoo people have gone.
<whitequark>
it also requires systemd
<whitequark>
well, irrelevant for our use case, since if I'd use it, I would replace conda with it, not the entire system
<rjo>
do they never hit the problem that the kernel does not fit their picture?
<whitequark>
it doesn't?
<whitequark>
ah yeah, the entire system doesn't depend on kernel version, right
<whitequark>
that's funny
<rjo>
kernel gets a hash, only one kernel can run, a set of separately packaged kernel modules get a hash, only one of those can be used, user space depends on specific version, only one userspace can run etc
<whitequark>
no, they allow 'dynamic linking' for kernel apparently
<whitequark>
so it's ideologically sloppy instead
<rjo>
and there are also these non-path dependencies. like all kinds of protocols and other ABI/APIs that are not exposed through path hashes, X11/wayland/ipython-kernel/...
<whitequark>
yeah
<whitequark>
what I really want is transactional dpkg
<whitequark>
this doesn't require any of the stupid path tricks
<rjo>
conclusion: Nix is Nix for me.
<whitequark>
mhm, I wouldn't discard it so quickly, it's a much saner thing than conda, especially if we don't run the whole OS
<rjo>
i'll grant it the title "multiarch done right"
<rjo>
whitequark: about the conda version thing: let's downgrade. what do i have to watch out for if i'd do that on the buildslaves?
<rjo>
whitequark: also did you only expose udp 60000 and 60001?
<whitequark>
there was some other conda bug exposed by that, you can find it in channel logs
<whitequark>
yes, do you need any other ports?
<whitequark>
actually, it's only 60001
<rjo>
that only allows one mosh session for all of us.
<rjo>
that is the --output problem that they are mumpling about
<rjo>
mumbling
<whitequark>
yes
<rjo>
ok. let's leave that for tomorrow. g'nite.
<whitequark>
o/
fengling has joined #m-labs
fengling has quit [Ping timeout: 272 seconds]
fengling has joined #m-labs
sb0 has joined #m-labs
<sb0>
qt5 brings embedding, plus improvements to the docking system
rohitksingh has joined #m-labs
<sb0>
whitequark, I'm not sure how the embedding works internally for Qt, but Windows does have a solution to embed a window into another
<sb0>
like X11
<sb0>
so Qt4 inside Qt5 is theoretically possible as well
<sb0>
but note that matplotlib has a Qt5 backend, which is what I used last time for embedding. writing the applet-side Qt4 embedding code is potentially difficult, with the Qt bugs and all
bentley` has quit [Ping timeout: 245 seconds]
<sb0>
as for qt patches. all I'm thinking about are bugfixes or maybe additions of small features that would be backward-compatible (e.g. support for side-by-side docks in floating windows, if that's ever needed)
<sb0>
and merged upstream qt
<sb0>
can't we get everyone in anaconda to use our package?
<sb0>
wow, the qt docking system doesn't have the applet bug
<sb0>
I'm impressed
sb0 has quit [Quit: Leaving]
<GitHub119>
[artiq] sbourdeauducq created qtdocks (+1 new commit): https://git.io/vgyyD
<GitHub119>
artiq/qtdocks daf49ef Sebastien Bourdeauducq: gui: rough conversion to the Qt docking system
sb0 has joined #m-labs
<sb0>
it also doesn't have the hidpi bugs
rohitksingh has quit [Ping timeout: 240 seconds]
rohitksingh has joined #m-labs
<cr1901_modern>
I've been led to believe in the past that Qt is exceptionally buggy. I can only think of one offhand tho that may or may not be fixed in Qt 5. pyqtgraph, on the other hand...
<cr1901_modern>
(That bug I'm thinking of is O(n^2) complexity for listing directories on Windows
mumptai has joined #m-labs
<sb0>
ok, found two bugs already
<sb0>
but it's still not as bad as pyqtgraph
<sb0>
and one of them might be a gnome bug anyway
<GitHub186>
[artiq] sbourdeauducq pushed 1 new commit to qtdocks: https://git.io/vgy52
<GitHub186>
artiq/qtdocks 3c12c13 Sebastien Bourdeauducq: gui: dock state save/restore
<sb0>
now I just hope (Py)Qt won't give me crap for doing nothing reasonable (e.g. put it into a menu) with the QAction...
<whitequark>
that is actually reasonable
<sb0>
though I can put it into a menu and hide it
<whitequark>
that's how QActions are supposed to work
<whitequark>
though I fail to see the point in your case
<sb0>
whitequark, docks don't give you a way to tell you when the user closes them
<sb0>
but they give you a QAction for closing and opening the dock, and the checked state of that QAction is synched with the user closing the dock (with the cross button)
<whitequark>
why do you need to put it into a menu?
<sb0>
my idea is to use that QAction, and hook its checked state to detect when the user closes the dock with the cross button
<sb0>
I don't know, does (Py)Qt handle correctly a QAction that is referenced nowhere except in a one signal connection for its checked state?
<whitequark>
rjo: you don't have artiq anywhere at the time when the template is evaluated
<whitequark>
well, no, you do have it in the git repo
<whitequark>
correction
<rjo>
exactly.
<whitequark>
you don't have the ability to `import artiq`
<whitequark>
because its dependencies are not installed
<whitequark>
this is actually a second problem with that
<whitequark>
ideally I would not have the build: artiq dep *at all*
<rjo>
pretty sure versioneer can still run.
<whitequark>
but you cannot `import artiq`
<whitequark>
it will fail
<whitequark>
therefore, you do not have artiq.__version__
<rjo>
why? all it does is import versioneer.
<whitequark>
oh, it does now?
<rjo>
well yes. that was one of the reasons to do it.
<whitequark>
ok
<whitequark>
then just remove the dep
<rjo>
which dep?
<whitequark>
since the fact that we needed to install dependencies to `import artiq.gateware` was the only reason to have `build: - artiq` in meta.yaml in the first place
<whitequark>
in gateware packages
<rjo>
but still. i would want to runtime depend on the correct artiq.
<whitequark>
that's what the run: - artiq dep is for
<whitequark>
look below.
<rjo>
yes. replace that three-line-wonder by the same stuff that would get the version string in the main artiq meta.yaml from versioneer.
<whitequark>
nah, who cares, it works
<whitequark>
and it does create a dependency on correct artiq
<rjo>
until conda decides that these strings should be composed differently from all the ingredients.
<rjo>
e.g. make that py_ a py35_ ...
<whitequark>
it will do the exact same thing in main artiq package
<whitequark>
ah
<whitequark>
anyway, that is separate from the build dep
<whitequark>
which I've just removed
<whitequark>
if you want, you can make it use versioneer
<GitHub185>
[artiq] whitequark pushed 1 new commit to master: https://git.io/vgyFg
<GitHub185>
artiq/master 1cfca74 whitequark: conda: remove spurious artiq dep from gateware packages.
<whitequark>
though I fail to see what it would improve
<rjo>
yes. i'll leave it for now. but conceptually it would be nice to use versioneer for the conda packages as well.
<whitequark>
you have the same issue with py_ with versioneer.
<whitequark>
I'll tell you more
<whitequark>
conda does not control whether py_ appears there or not
<rjo>
i took another kc705 from boulder and will have that for myself soon. but otherwise ok to mess with the kc705 on that machine? i'll give a warning when i do.
<rjo>
whitequark: afaict conda build --output is not all that reliable. if another build intervenes, you get empty versions. could we make buildbot use those slaves more exclusively?
<rjo>
as in start-to-end exclusivity?
<whitequark>
er, no/
<whitequark>
--output should be reliable.
<whitequark>
there's no reason it can not reliable
<whitequark>
it only should care about the current checkout.
<rjo>
whitequark: i suspect it depends on some state left over from the actual (last) build. at least it did at some time.
<whitequark>
buildbot interlocks function on the step level, not build level...
<rjo>
yes. that's why i suspect that the other build started in the same slave while get_output of a just completed build is run caused this problem.
<whitequark>
which problem exactly?
<whitequark>
I get empty versions all the time without intervening builds
<rjo>
are you looking into locking the build slaves so that at least --output always immediately follows its build?
<whitequark>
that's not possible
<whitequark>
although, no, hang on, this of course is
<whitequark>
the word "locking" misled me
<whitequark>
rjo: actually, there is no need to do so, buildslaves are already restricted to 1 concurrent job
<cr1901_modern>
rjo: I'm sorry to ask this, but in your ticker class, what does "dynamic" actually do? I interpreted it as "the power at which we start modifying what's displayed on the magnitude and offset label"
<rjo>
whitequark: but it switches to another job between steps.
<whitequark>
rjo: that seems unlikely
<whitequark>
can you show me with which three steps exactly that happened?
<rjo>
cr1901_modern: if log10(delta/center) < dynamic it should rewrite numbers.
<whitequark>
I poked around when we discussed that above and couldn't
<rjo>
whitequark: #107 and #124 both ran on the -2 slave.
<whitequark>
oh
<rjo>
both seem to start build at the same time, the #107 build finishes first at 22:54 then the 124 build at 23:18
<whitequark>
*facepalm*
<rjo>
right after the build both are supposed to do get_output
<whitequark>
I gave it the wrong option in the wrong place.
<whitequark>
you are completely correct of course.
<whitequark>
one moment.
<rjo>
cr1901_modern: dynamic as in "dynamic range" ~ "requires many digits to represent a small variation on top of a large number"
<GitHub34>
buildbot-config/master 4875e82 whitequark: Actually restrict the number of parallel jobs to 1.
bb-m-labs has quit [Quit: buildmaster reconfigured: bot disconnecting]
bb-m-labs has joined #m-labs
rohitksingh has quit [Ping timeout: 276 seconds]
evilspirit has quit [Ping timeout: 240 seconds]
evilspirit has joined #m-labs
<rjo>
sb0: why did you assign me to #251? you wanted all the subprocesses, and that's your branch.
rohitksingh has joined #m-labs
evilspirit has quit [Ping timeout: 256 seconds]
<cr1901_modern>
rjo: Ack. I understand now.
<sb0>
whitequark, i think you can simply disconnect the fan
<whitequark>
ok
<sb0>
it has a connector iirc, and the artiq design doesn't dissipate that much power in the fpga
<sb0>
just do a summary check of the temperature of the FPGA after a while, but it should be fine I think
<whitequark>
ok
<sb0>
apparently, executing sys.executable is problematic on windows
<sb0>
funny...
<sb0>
it results in FileNotFoundError, which is ridiculous
<whitequark>
wtf
<sb0>
after working around this, the main GUI process crashes when the applets RPCs its embed function... bah
<sb0>
(freezes...)
<sb0>
rjo, you said you'd update the buildbot to run the test according to the documentation I have written in the newtesting branch
<sb0>
which is what #251 is about, not so much subprocesses
<rjo>
the bug is about the subprocesses causing some common unittest invocations to break. i said that we need to fix the buildbot when that branch is merged. did you check that the tests run etc? if you think you have no time to adjust the buildbot when you merge the branch, sure i can do it for you.
<rjo>
but if sys.executable does not work on windows, this might be something worth fixing before.
<sb0>
the tests run on my local machine, yes
<sb0>
linux with distribution packages
<GitHub51>
[artiq] sbourdeauducq pushed 2 new commits to master: https://git.io/vgS8L
<whitequark>
can you focus that window and close it using alt+f4?
<sb0>
maybe retrieve it from the layout
<whitequark>
yes, that's what I meant
<sb0>
it's a hack, but the alternative is the qt patch
<whitequark>
I'm not sure the patch can be written
<whitequark>
my point is
<whitequark>
if you can close the window in ways other than pressing the close window, then you can only really look at a sequence of show and hide events
<whitequark>
so I know for a fact that on Linux, my WM allows me to send a window a hide event without any interaction with decorations whatsoever
<whitequark>
and I think with tool windows on Windows, which it seems to use, the situation is the same
<sb0>
yeah, alt-f4 will close only the dock
<whitequark>
i.e. Windows draws decorations, and the X button on the decoration just sends the hide event
bb-m-labs has quit [Quit: buildmaster reconfigured: bot disconnecting]
bb-m-labs has joined #m-labs
<whitequark>
so hooking up the close button is not a solution, and this is why you cannot
<sb0>
First it sends the widget a QCloseEvent. The widget is hidden if it accepts the close event. If it ignores the event, nothing happens. The default implementation of QWidget::closeEvent() accepts the close event.
<sb0>
sometimes, docks that are floating fail to reappear on state restore
<sb0>
but this seems to be a gnome-related problem
<sb0>
(gnome3, compositing...)
<whitequark>
actually, qt's source isn't half bad
<whitequark>
I can probably fix qt for you myself after all
<whitequark>
thought it'd be worse but no. I guess after SolveSpace and LLVM there's fairly little in real-world C++ that would really scare me; Qt looks downright boring
<cr1901_modern>
What's scary about SolveSpace?
<cr1901_modern>
Wait... I'm sorry, don't answer that
<whitequark>
there's nothing particulary scary about SolveSpace except the lengths it goes at places to do everything like C does
<whitequark>
the scary part was adding STL to a codebase that thinks it's OK to just memset random shit
<whitequark>
I've spent *weeks* debugging segfaults and learning about STL design constraints
<cr1901_modern>
Solvespace to me feels (felt?) like "C with classes" for the most part.
<GitHub115>
[artiq] sbourdeauducq pushed 2 new commits to qtdocks: https://git.io/vgSzH
<GitHub115>
artiq/qtdocks d81ce15 Sebastien Bourdeauducq: gui: Qt wants a central widget in the main window, use explorer
<GitHub115>
artiq/qtdocks cd73271 Sebastien Bourdeauducq: gui: save/restore main window geometry
<whitequark>
yes, original SolveSpace was written like this
<whitequark>
I've put a fair amount of std::vectors and std::strings into it though already
<whitequark>
and it's not like I'm going to slow down
<GitHub142>
[artiq] sbourdeauducq pushed 1 new commit to qtdocks: https://git.io/vgSgO