<_whitenotifier-c>
[Glasgow] peteut commented on commit 1d8e1d07de8714ad83ff8719c9e8b0b3f7214030 - https://git.io/fharb
<tnt>
azonenberg: btw, what's the status of debugging your link flapping issues ?
<azonenberg>
tnt: i built this board to get more details on the actual system behavior
<azonenberg>
i still dont know what's causing it, i've tried swapping a few components i suspected
<azonenberg>
i swapped the phy chip itself with a newer die revision
<azonenberg>
i found an older board i did a few years ago with the same phy that works and links up in gig
<azonenberg>
trying to do some a/b testing between them
<azonenberg>
also integralstick uses the same phy, i have one board assembled but not tested b/c the breakout with the rj45 on it hasnt shipped from oshpark yet
<azonenberg>
So fingers crossed that IT works
<azonenberg>
i also have a support case open with MCHP, the engineer hasn't yet found anything i didn't already know about and rule out
<azonenberg>
the failing and working board are super similar
<azonenberg>
both have a ksz9031 connected to the same magjack
<azonenberg>
powered by a ltc3374
<azonenberg>
and talking to a xilinx 7-series part
<azonenberg>
(also bedtime, i have an 0730 client call in the morning)
<tnt>
running at the same voltage ?
<azonenberg>
avddh is 3.3 on the failing board and 2.5 on the working one i think
<azonenberg>
pretty sure io is 1.8 on both
<azonenberg>
slightly different decoupling layout is about the only thing i can think of
<quinor>
whitequark: a bit of OT, heard you've connected two
<quinor>
PCs using pcie, how does this work?
<quinor>
also, hello (I should have started with that probably)
<whitequark>
I have not done that
<whitequark>
in theory, you can use a non-transparent bridge for this
<quinor>
it sounded so ridiculously I just had to know if it really works, thanks :)
<whitequark>
you can definitely do it
<whitequark>
the main problem is you don't get cache coherency
<whitequark>
so, people are mostly using it for things like "using a single network card with two PCs"
<whitequark>
PCIe is closer to Ethernet in many respects than PCI
<q3k>
it's also usable for RDMA
<q3k>
as long as you take care of cache coherency, which is generally the case with PCIe based RDMA
<whitequark>
but i wanna make linux do nccNUMA ;w;
<whitequark>
actually
rohitksingh has quit [Ping timeout: 240 seconds]
<whitequark>
q3k: i should be able to run two linuxes in the shared RAM by partitioning it and making them use ring buffers to communicate
<whitequark>
wait. i've invented, er
<q3k>
i spent quite a while thinking about this
<whitequark>
whatever is the opposite of NUMA
<whitequark>
i've invented regular ass networked PCs
<whitequark>
(with rDMA)
<whitequark>
and everyone does that already
<q3k>
iirc there's also single image linux hacks that work with non-cache-coherent RDMA
<q3k>
ie over infiniband
<q3k>
but that does not give you NUMA
<q3k>
this is also relevant to my silly idea of running some sort of cache coherency protocol over udp multicast
<q3k>
so i can make a supercomputer out of RV901Ts
<q3k>
by just stacking them on cheap gbps switches
<q3k>
just need to find an rv64 core with privileged spec that will fit on the spartan LX15
<Ultrasauce>
that is definitely the most cursed thing i'll read today
<ZipCPU>
q3k: And here I thought getting a multiprocessing ZipCPU running on the Spartan 6-LX9 was a challenge
<q3k>
ZipCPU: realistically, can we get zipcpu to run linux?
<q3k>
ZipCPU: does it already run linux?
<ZipCPU>
q3k: It was designed to run Linux, so Yes
<q3k>
ZipCPU: if i can fit it with some cache and an sdram controller on an LX15 i'm totally doing that
<ZipCPU>
That said, I haven't integrated the MMU, so you'd need to run the embedded linux on it only
<ZipCPU>
(While it was designed for that purpose, I haven't gotten that far in any of my implementations ...)
<q3k>
i want real linux
<q3k>
not uclinux or linux-nommu or however it's called this month
<ZipCPU>
If you drop the caches, it might be doable. The cache integration is part of the difficulty
<ZipCPU>
But it would still require some work to make it happen
<q3k>
whitequark: linux-boneless when
<whitequark>
q3k: the only supported language will be rust :P
<whitequark>
well, maybe, maybe not
<tnt>
ZipCPU: what kind of size/fmax does zipcpy get on an ice40 ? (assuming minimal config)
<tnt>
-y+u
<ZipCPU>
Using an iCE40, you only get a minimum configuration ;D Nothing else fits
<ZipCPU>
As for fmax, I can run at (roughly) 50MHz on the hx8k's
<ZipCPU>
On the TinyFPGA BX (an lx8k) I can only get 25Mhz
<tnt>
tx
<kc8apf>
quinor: I've connected 9 machines via PCIe NTBs
rohitksingh has joined ##openfpga
kiboneu is now known as cnomad
<ZipCPU>
tnt: There's more to the story. Specifically, your performance will not be determined by fmax since I had to strip the pipelining out of the CPU. Indeed, it's even worse than that since the cache is gone too, so your CPU speed is determined by the I/O on the iCE40 not by fmax
<tnt>
ZipCPU: sure but often my interest by fmax is more motivated by other part of the system that need a specific frequency and desire to avoid clock crossing as much as possible and also only 1 pll so I can't generate that many clocks in the first place either :p
<ZipCPU>
k, that makes sense
<quinor>
kc8apf: curious, were you able to do that on consumer hardware?
<kc8apf>
I was using proprietary boards but everything was based on commodity parts
<kc8apf>
It _should_ work on consumer hardware
<quinor>
nice
<kc8apf>
but there's a fair amount of work involved in getting the OSes to do what you want
<quinor>
figured that one out, don't think I'd be able to do that though
<kc8apf>
Main component was PLX PCIe switches configured as NTBs
<kc8apf>
opening up memory decoding across the NTBs is pretty straightforward but that's only a small part of the story
<kc8apf>
you'll generally want to provide some sort of messaging scheme to deliver interrupts, etc
<quinor>
sounds like hell to make software stack work
<kc8apf>
wasn't easy
<kc8apf>
and then it ends up being fragile too
<kc8apf>
we connected 8 machines to 1 "headend"
<kc8apf>
the headend was in control of all the memory windows and peripherals
<quinor>
also, any idea as for where to source pcie switches (this time I "just" wanna have more pcie x8 or x16 slots in a machine)?
<kc8apf>
if it went down for any reason, it was roughly the equivalent of yanking a PCIe card from a running machine
<whitequark>
hahahahaha you're fucked
<whitequark>
all pcie switch companies are evil
<whitequark>
i've looked into it
<quinor>
wouldn't ask if it was easy
<whitequark>
they a) require you to sign an eye wateringly wide in scope nda
<quinor>
kc8apf: pcie is supposed to be hot-plug
<whitequark>
b) probably will just ignore you anyway
<whitequark>
don't bother with broadcom
<kc8apf>
quinor: only if you implement hot-plug
<quinor>
works with most stuff on linux if you unload the driver module before unplug
<quinor>
else, kernel panic
<kc8apf>
when there is no physical unplug event, something needs to generate that event based on noticing the other end of the logical connection is down
<whitequark>
kc8apf: yeah what's up with that?
<whitequark>
why is there no module that generates hotplug events based on pcie link status?
<whitequark>
is it because root ports can't generate interrupts on plug events?
<whitequark>
(or can they?0
<kc8apf>
link down _could_ trigger an event
<kc8apf>
but NTBs are weird
<whitequark>
it doesn't work for any devices IME
<whitequark>
AER... reports things, occasionally
<kc8apf>
depending on configuration, each side can maintain the link independently
<whitequark>
but thunderbolt does most of its stuff via gross ACPI hacks
<kc8apf>
it's been a while since I read the PCIe hotplug spec
<kc8apf>
IIRC the electrical hotplug capability is only about not damaging components. It doesn't trigger OS-level events.
<whitequark>
yes
<whitequark>
but you can look at LTSSM status
<kc8apf>
Expectation is that software has already done all the work _before_ the physical link is done
<whitequark>
it's there in config space, for one
<whitequark>
hrm
<kc8apf>
this is likely something no one has just never implemented
<whitequark>
I mean, at worst you could poll link status
<whitequark>
it's dumb but it would definitely work
<kc8apf>
PCIe hotplug is typically used for cases where you want to replace a board w/o downtime. So, you push a button to tell the OS to prepare the slot for removal. The OS does whatever it needs to and then turns on a light to say it's ready. Then you yank the card.
quinor has quit [Quit: Konversation terminated!]
<balrog>
kc8apf: how does it work for thunderbolt?
<balrog>
where someone can just yank the cable
<balrog>
(still not pleased that the thunderbolt spec is secret)
<kc8apf>
balrog: magic and ACPI hacks
<balrog>
kc8apf: yeah, seems like hacks upon hacks
<kc8apf>
I recall TB dev spending a lot of time on figuring out how to make surprise removal events work because PCIe hotplug isn't made for that
m4ssi has quit [Remote host closed the connection]
<whitequark>
balrog: i've investigated it in depth
<whitequark>
ask me if you want to know
<whitequark>
but you don't
<whitequark>
you really don't want to know how the TBT sausage is made
rohitksingh has quit [Remote host closed the connection]
<whitequark>
balrog: oh they FINALLY fixed pciehp to be not shit
<whitequark>
thank god
<whitequark>
that... that might explain why TBT is more stable now
<whitequark>
wow, that page is nothing but good news
<whitequark>
i'm impressed
<sxpert>
kc8apf: nah, pcie hotplug should be seamless, no button needed. that's implemented in thunderbolt. in practise, it doesn't seem to work well for typical pcie hardware...
<whitequark>
sxpert: are you actually aware of how it's implemneted in thunderbolt?
<sxpert>
I believe it's using the usb bits for detection, which should also work with card style pcie which also has usb
<whitequark>
'card style pcie'?
<kc8apf>
PCIe edge connectors do not have USB
<kc8apf>
some companies use A31/32 but that is far from standard
<whitequark>
it's also not relying on USB for negotiation
<whitequark>
some of it is done via USB PD and some via Thunderbolt NHI
<kc8apf>
PCIe hot-plug specification lays out the button, etc. I've not seen any spec defining what should happen for surprise removal
<kc8apf>
per the LWN article shared earlier, Facebook ran into this with external trays and yanking the interconnect cable
<kc8apf>
they came up with ways to make Linux not die a horrible death
<sxpert>
pcie has smbus on 5/6 (A and B), which may also be somehow used
<kc8apf>
I know Google runs a hard reset line through their external PCIe cables which causes a host reboot on disconnect
<sxpert>
also A1 -> B(17/31/48/81) may be used for card presence
<kc8apf>
sxpert: nope. those are often unimplemented. when they are, it's non-standard
<sxpert>
that one is said "must connect"
<kc8apf>
yes, presence detect works
<kc8apf>
are you researching this right now?
<sxpert>
yeah
<kc8apf>
I spent a decade working on PCIe in motherboard design at Apple and Google
<sxpert>
ah well, I bow to the master thn
gsi_ has joined ##openfpga
<sxpert>
imho, if you need a button, then it's not "hot plug".. it's half baked
<whitequark>
kc8apf: oh, nice
<whitequark>
if i was actually able to get any docs for the switches i want to use, i'd pester you with questions
<whitequark>
but, as you probably already know, fucking broadcom
<kc8apf>
yeah
<kc8apf>
if I still had them, I'd just give them to you
<sxpert>
whitequark: "fucking broadcom" is the real name of the company ;)
<whitequark>
kc8apf: i assume you don't have any microsemi switches either
<whitequark>
er, docs for them
<kc8apf>
nope. Google was entirely a PLX shop
<kc8apf>
whose design did Microsemi end up with?
gsi__ has quit [Ping timeout: 250 seconds]
<kc8apf>
there used to be only PLX and IDT but then they were merged
<whitequark>
Switchtec
<whitequark>
er, wait, no
<whitequark>
not sure where they got the switches, actually
<kc8apf>
these are very similar to PLX part #s
<whitequark>
that's true
<kc8apf>
ah, it is the IDT design
<kc8apf>
PMC-Sierra bought IDT then Microsemi acquired PMC-Sierra
<whitequark>
you can get full datasheets from IDT but only for their Gen2 devices
<whitequark>
and unfortunately Gen2 is too slow for my goals :S
<kc8apf>
I expect the design didnt' change much
<whitequark>
yeah but I can't even get a pinout for any of the Microsemi Gen3 devices
<kc8apf>
:(
<sxpert>
whitequark: guess you need to sign NDAs and buy a million ;)
<sxpert>
and possibly be a member of the PCIe consortium or something (more millions $)
<whitequark>
sxpert: i do not guess. i am well aware of what specifically their requirements are
* sxpert
goes hide into a corner, being confronted by the 1st degree team
<whitequark>
and if it was -just- an NDA that would not be so much of an issue
<whitequark>
but that was the single most draconic NDA I have ever heard of
<kc8apf>
sxpert: you've wandered into a room filled with literal experts on large parts of computing hardware design and development
<sxpert>
I was trying to insert some comic relief, guess you guys are not receptive
<kc8apf>
did not come across as humor
<whitequark>
yeah
<kc8apf>
especially after you were researching PCIe hotplug earlier
<balrog>
> single most draconic NDA I have ever heard of
<balrog>
sounds bad
<whitequark>
i cannot imagine anyone signing that other than on behalf of a shell company created specifically to hold that agreement
<whitequark>
or something like that
<whitequark>
they're insane
<kc8apf>
big company legal would redline a bunch and send it back
<whitequark>
right, but you have the power
mumptai has joined ##openfpga
<kc8apf>
I wonder what they'd do if you sent it back with edits
<sxpert>
"can potential customer can gobble me up with their wads of cash ?"
<davidc__>
Eh, I'd redline it even as an individual. I've been surprised by how successful that has been in the past
<davidc__>
(Especially if you can make a case for potentially buying a decent volume of their parts)
<sorear>
##openfpga should be able to make switches soon, although gen3 x8 won’t be cheap
<whitequark>
xilinx?
<sorear>
would have to be for gen3. obviously this increases the amount of integration work the person doing this would be faced with
<whitequark>
multilane pcie is beyond my understanding even
<gruetzkopf>
iirc it's just bytewise striping
<gruetzkopf>
fairly simple if you can assume the relative length of the lanes doesn't change while the link is up
<whitequark>
gruetzkopf: nono
<whitequark>
well, i guess "beyond my understanding" is too harsh
<whitequark>
i understand how it works
<whitequark>
i don't know how to implement it
<whitequark>
the easy hard part is lane deskew
<whitequark>
the hard hard part is the parser
<whitequark>
theproblem is that SDP and STP (iirc these are the right names) can appear on any lane although with certain conditions
* gruetzkopf
rereads spec
<gruetzkopf>
okay, after link idle it'll always be on lane0. with up-to-4-lanes, it'll always be on 0. more generalised it's on a lane mod 4 = 0
<gruetzkopf>
-> if your internal datapath is 32bit wide it should just workTM
<sxpert>
gruetzkopf: it sends one byte througn l0, the next one through l1, and so on then goes back to l0 ?
<gruetzkopf>
well, it'll send bytes on all lanes simultaneously, but yeah
<sxpert>
I see
<whitequark>
gruetzkopf: there's another issue i recall
<whitequark>
like you could have SDP and STP simultaneously
<whitequark>
at the same clock?
<sxpert>
you'll need a bunch of serdes, one per channel I suppose, that's gonna be a huge fpga
<whitequark>
well, symbol clock
<gruetzkopf>
yes
<whitequark>
so, even with that, lane mod 4 = 0?
<whitequark>
wait, hm
<whitequark>
SDP and STP are all 32 bit aligned right?
<whitequark>
er
<whitequark>
DLLP and TLP I mean
<whitequark>
32-bit aligned and multiple of 32 bit in size
<whitequark>
gruetzkopf: hm, I think I figured it out
<whitequark>
the idea is to run DLLP parser and TLP parser -in parallel'
<whitequark>
each should skip everything until SDP or STP when it's in the idle state
<whitequark>
but you could still only have one instance of the parser
<whitequark>
that's clever
<gruetzkopf>
yeah, i don't know how they managed that despite intel being involved
<sxpert>
one is stared by SDP and the other by STP ?
<sxpert>
started even
<gruetzkopf>
even the x32 max link width is to get around another edgecase iirc
<whitequark>
which?
<gruetzkopf>
oh, misread
<gruetzkopf>
disregard
<gruetzkopf>
"The longest sequence of width negotiation consists of an upstream component (downstream port), which supports x16, x8, x2, x1 connected to a downstream component (upstream port), which supports x32, x12, x4, x1. Step 1 would attempt to create a x16 Link, step 2 a x12 Link, step 3 (first pass) a x8 Link, step 4 (first pass) a x4 Link, step 3 (second pass) a x2 Link, step 4 (second pass) a x1 Link."
<gruetzkopf>
x12!?
<gruetzkopf>
okay, seems to be the only non-power-2 link wifth allowed
<whitequark>
yes.
<whitequark>
x24 used to be a thing.
<whitequark>
in pcie 1
<gruetzkopf>
hm
<whitequark>
iirc there was a reason they took it out but i can't recall which one it was
<whitequark>
no one really implements x12 either
<gruetzkopf>
not in the release of the 1.0 spec i have (dated 2002-04-29)
<whitequark>
hm
<gruetzkopf>
i mean sure, it's multiple-of-32-payload-bits-per-symbol
<whitequark>
x2 isn't
<whitequark>
ah, nvm
<whitequark>
i definitely remember something about x24
<whitequark>
whether it's a power of 2 is irrelevant
<whitequark>
whether it's a multiple of 4 is far more interesting
<whitequark>
non-multiple-of-4 lane counts other than 1 and 2 would probably be unreasonable to support
<sxpert>
the byte distribution module would be unwiedly
<whitequark>
have you actually thought about the implementation, or are you baselessly speculating
<sxpert>
I'd have the packets as an array of 32 bit values, and something that would distribute them between lanes/4 modules, the lane0-3 module also being able to handle 1 2 and 4 lanes
<tnt>
kbeckmann: btw, if you can spare a PCB, I'd gladly buy one off from you :)
<sxpert>
still 12 lanes complicates the matter because there's a 3 factor in there so you need a counter instead of a shift
<kbeckmann>
tnt: I built 2 complete boards and have one partially assembled (ordered too few components). If you're going to FOSDEM I can give it to you there, otherwise please let me know your address in privmsg and I'll send a board to you! I'm currently verifying that it works at all :).
<tnt>
kbeckmann: yeah, I'll be at fosdem, in the SDR room all sunday at least. (unless something really bad happens until then ...)
<kbeckmann>
awesome! see you there, i'll bring an assembled board for you.
<tnt>
great, thanks !
<kc8apf>
sxpert: if you are trying to learn, ask questions
Flea86 has joined ##openfpga
<azonenberg>
My guess as to why there's no longer x24
<azonenberg>
is the physical size of the connector :p
<azonenberg>
x16 is already massive
<whitequark>
azonenberg: x32 exists
<whitequark>
you'd think they would ditch x32 and not x24
<whitequark>
amusingly, iirc no actual *implementations* of x32 (and x12) exist
<whitequark>
what you see as "x32" is usually riser cards with a proprietary layout
<Richard_Simmons>
is x32 any bigger than the vesa slots that briefly existed?
<Richard_Simmons>
I can't recall if I ever laid hands on a mobo with it, if I did it was a decade+ ago.
mumptai has quit [Remote host closed the connection]