sb0 changed the topic of #m-labs to: https://m-labs.hk :: Logs http://irclog.whitequark.org/m-labs :: Due to spam bots, only registered users can talk. See: https://freenode.net/kb/answer/registration
wishi14 has joined #m-labs
wishi14 has quit [Remote host closed the connection]
<sb0> hartytp: so the DACs are behaving themselves? you get sync between (D)RTIO and the RF output?
sb0 has quit [Quit: Leaving]
SmokedCheese27 has joined #m-labs
SmokedCheese27 has quit [Killed (Sigyn (Spam is off topic on freenode.))]
mpeterson1 has joined #m-labs
mpeterson1 has quit [Killed (Sigyn (Spam is off topic on freenode.))]
sb0 has joined #m-labs
mauz555 has joined #m-labs
mauz555 has quit [Ping timeout: 252 seconds]
microserver_19 has joined #m-labs
microserver_19 has quit [Remote host closed the connection]
sb0 has quit [Quit: Leaving]
noplamodo has joined #m-labs
noplamodo has quit [Remote host closed the connection]
plat_24 has joined #m-labs
plat_24 has quit [Remote host closed the connection]
rohitksingh has joined #m-labs
Infininight28 has joined #m-labs
Infininight28 has quit [Remote host closed the connection]
rohitksingh has quit [Quit: Leaving.]
Crashjuh16 has joined #m-labs
Crashjuh16 has quit [Remote host closed the connection]
<GitHub-m-labs> [artiq] sbourdeauducq commented on issue #1007: @whitequark can you give us an update? https://github.com/m-labs/artiq/issues/1007#issuecomment-425702032
rohitksingh has joined #m-labs
rohitksingh has quit [Quit: Leaving.]
rohitksingh has joined #m-labs
keks22 has joined #m-labs
keks22 has quit [Remote host closed the connection]
hartytp has joined #m-labs
<hartytp> sb0: no, I need to rebuild with SAWG and ran out of time yesterday
<hartytp> what I've done is to use the AD9154 + delay line to do a SYSREF eye scan and verify that it looks okay
<hartytp> sb0: can you do me a favour please? Can you do a DAC SYSREF eye scan using the HMC7043? I'd like to see how good it looks given the unterminated SYSREF DAC inputs
<hartytp> ideally, can you do this with the DAC clock at 2.4GHz (really easy to do, just change the HMC7043/HMC830 dividers and change the INTER_MODE DAC register to 0x03)
<hartytp> s /INTER_MODE /INTERP_MODE
<hartytp> sb0, _florent_: why are we using 9.375MHz for the SYSREF?
<hartytp> isn't it normally f_data/(K*(F/S))=600MHz/16=37.5MHz
<hartytp> (I suspect that the answer is that the data sheet is wrong, since it doesn't seem consistent with other references, but I wanted to check...)
rohitksingh has quit [Quit: Leaving.]
johnnyfive25 has joined #m-labs
johnnyfive25 has quit [Remote host closed the connection]
egeltje_ has joined #m-labs
egeltje_ has quit [Remote host closed the connection]
rohitksingh has joined #m-labs
sssolid2 has joined #m-labs
sssolid2 has quit [Ping timeout: 268 seconds]
rohitksingh has quit [Quit: Leaving.]
sb0 has joined #m-labs
<hartytp> sb0: I'm doing something dumb, but I can't see what. I added an eye scan to the JESD sync code to get a better look at possible SI issues with the delay line.
<hartytp> code is https://hastebin.com/gaxiyoruve.java (yes, I need to learn rust...)
<hartytp> using the HMC7043 to divide the 2.4GHz DAC clock down to 600MHz I see something like __ __ _ ____ ____ | \/ (_) ___| ___ / ___| | |\/| | \___ \ / _ \| | | | | | |___) | (_) | |___ |_| |_|_|____/ \___/ \____| MiSoC Bootloader Copyright (c) 2017-2018 M-Labs Limited Bootloader CRC passed Gateware ident 4.0.dev+1404.g05ae15be.dirty;standalone.without-sawg Initializing SDRAM... DQS initial delay: 110 taps Write leveli
<sb0> hartytp: iirc dac_sync() returns whether the detected sysref phase matches the one from the previous sync
<hartytp> yes
<hartytp> so that's plotting rotations
<hartytp> that looks fine
<hartytp> I think
<hartytp> i'll post the 1.2GHz one in a sec, which looks like junk
<sb0> yes, and when you don't meet s/h it will randomly return true or false
<hartytp> yes
<hartytp> so there should be patches with random combinations of '.' and 'x' and then patches with lots of '.' which are the valid regions
<sb0> does the period of the sync unstability match the theoretical value?
<hartytp> roughly
<hartytp> 11ps/tap should be 166taps/cycle at 600MHz
<hartytp> https://hastebin.com/hakukacimu.css here is a 1.2GHz sample
weust22 has joined #m-labs
<hartytp> okay, nope, I seem to get non-reproducible results between restarting the FPGA
<hartytp> should have assumed that would be the case
<sb0> well with that circuit you can probe what the FPGA is sending into the FF
<hartytp> same again after running artiq_flash firmware start
weust22 has quit [Remote host closed the connection]
<sb0> is that meeting s/h at the FF input?
<hartytp> 80 taps between working regions is what we expect for 1.2GHz
<hartytp> so that looks perfect
<hartytp> sb0: that's not the issue
<hartytp> I have one output of the phase shifter on a fast scope on persiost
<hartytp> persist
<hartytp> it looks good
<sb0> https://hastebin.com/hakukacimu.css might be caused by s/h not met at the FF input
<hartytp> I *think* the problem is SI
<sb0> note that ultrascale FPGAs have extremely unstable I/O timing, a "feature" that xilinx does not advertise much
<sb0> (across PVT + P&R runs)
<hartytp> sure, but as I said, I haven't seen any issues with the phase shifter output when I look at the other channel on a scope
<hartytp> same again after restarting the FPGA. this time it's junk
<hartytp> could be my signal levels being marginal so we randomly drift between working and not working
<sb0> what do you mean with "not seen any issue with the phase shifter output"?
<sb0> how do you check?
<hartytp> I mean that when I look at the DAC clock, ref clock and SYSREF all on a fast scope I haven't seen any missing SYSREF edges
<hartytp> or any times when SYSREF preiod changes
<sb0> if s/h isn't met, but the input pulse is long enough, it may cause shifting of SYSREF by one coarse cycle and a 'x' in your DAC sync scan
<sb0> ("coarse" = FF clock period)
<hartytp> yes but I would see that on the scope
<hartytp> e.g. trigger scope of sysref
<hartytp> put scope on infinite persist
<hartytp> and look at the refclk (150MHz) trace
<hartytp> if there's no blur on that then the FF is fine
<hartytp> hmm, nope that won't do it
<sb0> you need to output SYSREF from the FPGA and trigger on that
<sb0> (i.e. generate a new SYSREF based on the rtio counter just to trigger the scope)
dorian15 has joined #m-labs
<sb0> or trigger on the FF input, maybe
dorian15 has quit [Killed (Sigyn (Spam is off topic on freenode.))]
rohitksingh has joined #m-labs
<hartytp> okay, teed off the SYSREF FPGA output on the scope
<hartytp> triggered off that
<hartytp> looking at delay line output on persist
<hartytp> not jitter evident
<sb0> didn't that jitter only appear on some FPGA restarts?
<sb0> is the scope trace exactly the same with a trashed DAC sync scan and with a working one?
<hartytp> okay, I'm pretty sure it's SI
<hartytp> sometimes I get "no SYNC lock" errors
<hartytp> which, I think menas that sometimes it doesn't get an edge
<hartytp> damn
<hartytp> I need to add a proper biasing network but there is just nothing to solder too
<hartytp> vias are about 100um
<hartytp> no exposed ground plane
<hartytp> sb0 the other way I've checked that I don't see any FF S/H issues is to measure the difference between rising edges at the delay line output
<hartytp> (i.e. look at just that one output with the scope on persist, but offset the scope x-axis by one period)
<hartytp> and, again, I see nothing suspicious
<hartytp> what's odd is that the DAC SYNC rotations seem to have some structure/periodicity in them
<hartytp> as if I'd got the SYSREF period wrong
<hartytp> well, it's a mystery to me because 600MHz almost always seems to look perfect (and those few times it doesn't could have been user error -- my test setup is quite complex and fragile by now)
<hartytp> but, 1.2/2.4GHz basically never works
<hartytp> but, if it were just jitter/SI I would have expected to see something on the 600MHz scans
<hartytp> again, here is the 600MHz trace
<sb0> reflections maybe?
<hartytp> but, those should show up in the 600MHz trace
<hartytp> anyway, it would be helpful to see a repeat of this measurement with the HMC7043
<hartytp> it's pretty quick to do
<hartytp> I'm terminating my coax (50Ohm resistors to 100nF capacitors to ground to avoid interfering with the 600mV SYSREF bias that the DAC provides) where it joins the RTM but that's quite far from the
<hartytp> dac
<hartytp> maybe it's just noise, but that structure in the 1.2GHz trace seems odd as well...
Dieterbe has joined #m-labs
bkst has joined #m-labs
<hartytp> only thing I can think of is that there is more noise around when the DAC runs at higher frequencies so SI becomes more of an issue
<hartytp> anyway, that's about all I have time for today
<hartytp> let me know if you think of anything
Dieterbe has quit [Remote host closed the connection]
<hartytp> otherwise, it might be best to post my whole setup to greg and give him the unenviable job of trying to micro-solder the correct termination resistors on
<sb0> hartytp: can't think of anything now, but i'd suggest you try actually syncing the DACs with RTIO at 600MHz so there is at least one working demonstrator
bkst has quit [Remote host closed the connection]
<hartytp> sb0: yes, but it's not that easy
<hartytp> because to get 600MHz out of the HMC830 I need to use the output divider
<hartytp> so, we need to deterministically reset that
<sb0> ah right, of course it's sayma so there is some new reason why it won't work
<sb0> sigh
<hartytp> it's an easy fix for Sayma v2.0
<hartytp> route the HMC830 output to the FPGA as well as the ref clk
<hartytp> measure the phase of the DAC clock
<hartytp> and toggle the output divider value between 1 and 4 until we get the correct phase
<sb0> can we just stop using HMC garbage instead?
<hartytp> sb0: that's not fair
<hartytp> unless you know of a decent grade PLL whose internal VCO spans 600MHz 2.4GHz
<hartytp> then we have to have a divider after the VCO, which has to be reset
<hartytp> well, otherwise unless you can find a PLL where the output divider is inside the phase lock loop, in which case the divider reset time isn't important
<sb0> yes exactly, even xilinx can figure that out
<hartytp> well, that's a different market to the world of analog PLLs, which usually aren't designed to produce deterministic phases
<hartytp> anyway, if you can find something like that then great
<hartytp> otherwise, I don't think it's too much of an issue to have to reset the PLL. Don't we do that for the DRTIO transceivers?
fitzsim has joined #m-labs
<sb0> yes, because xilinx screwed up the phase slip feature
<sb0> i.e. it doesn't work with the elastic buffer bypassed (which is ironically when you'd need it the most), plus it has a remaining uncertainty of 1/2 clock cycle
<hartytp> anyway, if we can find a better PLL then great, otherwise I'm afraid that we're stuck with having to add some reset logic
<hartytp> anyway, for testing v1.0 I guess I can do the dumb thing of using a scope to look at the dac clock and ignoring all restarts where it doesn't have the correct phase
<hartytp> I'll do that
rohitksingh has quit [Quit: Leaving.]
<sb0> or keep the hmc830 bypassed and clock the board at 600MHz
<hartytp> true
<hartytp> wfty
<hartytp> wft
<hartytp> increased the number of points in the scan and 1.2GHz worked perfectly
<sb0> hmm
<sb0> and you are certain the the delay line output looks OK on the scope?
<sb0> at all times?
<hartytp> haven't seen any issue with it
<hartytp> 2.4GHz still borked
<hartytp> will try 1.2Ghz again and see if that was just a fluke
rohitksingh has joined #m-labs
<hartytp> hmmm...clocking at 600MHz doesn't help
<hartytp> does it
<hartytp> because where do I get the 150MHz from
<hartytp> okay, this is just so odd
<hartytp> every ~10 times I check the phase I see a rotation
<hartytp> hm...broken DAC sync logic at high clock frequencies?
<sb0> hartytp: divide the 600?
<hartytp> without introducing a phase ambiguity?
<hartytp> sb0: check out the latest data I posted to GitHUb
<hartytp> every 12th time I ask the DAC for its sysref phase it gives me a rotation when I clock it at 1.2GHz
<sb0> the 150M edges will always be aligned with the 600M edges, which is sufficient for demonstrating the hardware
<hartytp> true
<hartytp> okay, I'll do that then
<hartytp> obviously, it doesn't work with DRTIO, but we can fix that with a divider reset
<sb0> if you reconfigure it, the si5324 on the drtio master can eat the 600MHz directly
<sb0> take a kasli as drtio master, set it up for external clocking, and multiply N31 by 4
<hartytp> okay.
<sb0> that's all afaict. pretty easy.
<hartytp> but, I would still like to understand this behaviour as it will bite us again for Sayma v2.0 (interpolation is going to be required)
<hartytp> and, I can't see any way that SI could produce this periodicity
<hartytp> maybe it's some issue with the way we check the phase
<hartytp> or, worse, sync doesn't work with interpolation because of a hw bug
<hartytp> but, if that's the case then we need to know now
<hartytp> it would be really great to get a reproduction of this with the HMC7043 to rule out me doing something stupid
<sb0> hartytp: what part of the "FPGA restart" makes the bug appear and disappear?
<sb0> hartytp: the DAC is suspicious. is it redoing the DAC initialization?
<hartytp> it's almost always there for the 1.2Ghz and 2.4GHz cases
<hartytp> and almost never for teh 600MHz
<sb0> for kasli it's N32, not N31
<sb0> hm and replace this TC2-1TX which is rated for 300MHz only
rohitksingh has quit [Quit: Leaving.]
<sb0> or just blast it with enough input power that enough signal goes through, SI is acceptable, and nothing burns, if that works (there is no datasheet data above 300MHz)
<hartytp> sb0: I don't think that we need to demonstrate Sayma + Kasli
<hartytp> I'm happy that a local TTL + SAWG demo is sufficient
<hartytp> checking that DRTIO synchronises TTLs on Sayma with TTLs on Kasli can be done as a separate test
<hartytp> so, I'll just use the 600MHz reference and the HMC7043 to divide it down
<hartytp> I can probably clock my FF from the 600MHz without issue, but need to check that
<hartytp> or, if I have issues with that, I'll post-select
cosban18 has joined #m-labs
<hartytp> The thing that concerns me is these odd SYNC issues as no point doing a demo of TTL+RF sync if the DAC has some crap bug in it that means it will never be useful
<hartytp> larsc: any idea what could cause an issue like this https://github.com/sinara-hw/sinara/issues/583#issuecomment-425723389
<sb0> i'd take si5324 reprogramming and changing a balun any day over touching HMC trash
<hartytp> the numbers are the SYSREF phase in 11ps taps. I'm doing the eye scan by https://hastebin.com/asuxodobav.php
<hartytp> for 600MHz, no interpolation it looks fine
<hartytp> but for 1.2GHz (x2 interpolation) I get this odd issue where the SYSREF phase shows an error even if I don't touch the delay lin
<hartytp> line/sysref phase
<hartytp> but, the odd thing is that I see the errors with an almost fixed periodicity of 1 error per 10 times I query the phase
<hartytp> Can't see where that kind of periodicity can come from...
<hartytp> eye scan code is https://hastebin.com/vimivunazi.java
larsc_ is now known as larsc
<larsc> hartytp: what am I looking at?
cosban18 has quit [Remote host closed the connection]
<hartytp> larsc: have a look at this "eye scan"
<hartytp> this is using an ad9154 DAC and scanning the sysref phase w.r.t. the dac clock
<hartytp> numbers in the lock are the sysref delay w.r.t. the dac clock in 11ps taps (+ arb offset)
<hartytp> s /lock /log
<hartytp> so, in other words, for each delay we repeatedly put the DAC into single-shot mode and ask check for sysref realignments
<hartytp> 'x' means a realignment, '.' means no realignment
<hartytp> I would expect transition regions with lots of 'x's and good regions with lots of '.'s
<hartytp> or if the signal is junk (bad si) then a random scattering of 'x' and '.'
<hartytp> for 600MHz I see a nice trace with regions of 'x's and mainly '.'s
<hartytp> for 1.2GHz (2x DAC interpolation) I see that really odd behaviour
<hartytp> where about each 10-12 times we check the SYSREF phase we get a realignment
<hartytp> in a way that's not correlated with the phase of the SYSREF signal, or with the time between phase checks
<hartytp> I cannot figure out where that kind of periodic behaviour could come from
<larsc> and each line is the same check just being run at a different time?
<hartytp> right
<hartytp> each line is me printing the 'results' array
<larsc> my understanding would be that if you use interpolation all the link settings are still 100% the same, right?
<hartytp> yes
<hartytp> all I've done is change the DAC clock frequency and the INTERP_MODE register value
<hartytp> no changes to JESD core or to the SYSREF frequency
<larsc> and on the clockchip side, same vco frequency and just a different divider?
<hartytp> right
<hartytp> in this case, divider in the HMC7043
pierron has joined #m-labs
<hartytp> in any case, if there were some issue with clock noise or something I don't see how it could give an issue like this where we see periodicity in the phase checks; those aren't synchronous with any frequency on the dac
<hartytp> and adding the 333us delay between phase checks didn't change anything
<larsc> it could always be si issues and the repeating pattern due to some cross coupling of two frequencies
<hartytp> larsc: i considered that, but riddle me this
<hartytp> 1. in that case why do we not see any issue with the 600MHz scan
<hartytp> that looks perfect, so the SI would have to get worse at higher clock frequencies with the same SYSREF.
<hartytp> maybe ground bounce or something, but seems a stretch
<hartytp> 2. cross-coupling of which frequencies
pierron has quit [Killed (Sigyn (Spam is off topic on freenode.))]
<hartytp> the DAC is only sensitive to the phase of SYSREF once we arm it via spi. so the beat note we see would have to involve the frequency of the spi updates
<hartytp> and, on top of that, adding a delay between updates doesn't change anything
<hartytp> so, I can't figure out a plausible SI-related explanation for this
mulk5 has joined #m-labs
mulk5 has quit [Remote host closed the connection]
<larsc> ok, so one line is just running the sync check for the same delay setting multiple times?
<hartytp> yes
<larsc> and you've tried different values for clock::spin_us(333) and you get the same pattern?
<hartytp> Yes, I started without that at all
<hartytp> and got
<hartytp> then added it and got
<hartytp> each time, we get a pattern with a periodicity of 11 checks of the dac phase
<larsc> hm, that is really weired
<hartytp> implemented via write(ad9154_reg::SYNC_CONTROL, 0x1*ad9154_reg::SYNCMODE | 1*ad9154_reg::SYNCENABLE | 1*ad9154_reg::SYNCARM | 1*ad9154_reg::SYNCCLRSTKY);
<hartytp> clock::spin_us(1000); // ensure at least one sysref edge
<hartytp> let sync_status = read(ad9154_reg::SYNC_STATUS);
<hartytp> let realign_occured = sync_status & ad9154_reg::SYNC_ROTATE != 0;
<larsc> have you tried to split that into multiple writes to SYNC_CONTROL?
<hartytp> no
<hartytp> I suspect that the issue may relate to that code
<hartytp> maybe some issue with the register programming sequence at higher DAC clocks
<hartytp> (I didn't write that code and haven't poked at it yet)
<larsc> I'm pretty sure somebody asked about this sequence here before and whether that is OK
<hartytp> if you can recommend a programming sequence then I'll try it
<larsc> because the datasheet says it should be split
<hartytp> remind me where?
<hartytp> so, you wnat something like:
<hartytp> write(ad9154_reg::SYNC_CONTROL, 0x1*ad9154_reg::SYNCMODE);
<hartytp> write(ad9154_reg::SYNC_CONTROL, 0x1*ad9154_reg::SYNCMODE | 1*ad9154_reg::SYNCENABLE);
<hartytp> write(ad9154_reg::SYNC_CONTROL, 0x1*ad9154_reg::SYNCMODE | 1*ad9154_reg::SYNCENABLE | 1*ad9154_reg::SYNCCLRSTKY);
<hartytp> write(ad9154_reg::SYNC_CONTROL, 0x1*ad9154_reg::SYNCMODE | 1*ad9154_reg::SYNCENABLE | 1*ad9154_reg::SYNCCLRSTKY | 1*ad9154_reg::SYNCARM );
<hartytp> in that order?
<larsc> table 22
<hartytp> thanks
<hartytp> so how do you interpret that, given that all writes are to the same register? Do you mean something like the series of writes I posted above?
qwedfg28 has joined #m-labs
<hartytp> Maybe better would be to set the SYNC mode and enable the sync machine only once
<hartytp> okay, well I'll have a play with that and see if it helps
<hartytp> but, have to go now
<hartytp> larsc: thanks for the input, let me know if you think of anything else I can try, as this has got me scratching my head
qwedfg28 has quit [Read error: Connection reset by peer]
rohitksingh has joined #m-labs
S0071 has joined #m-labs
S0071 has quit [Remote host closed the connection]
Knirch_ has joined #m-labs
Knirch_ has quit [Remote host closed the connection]
atlas27 has joined #m-labs
atlas27 has quit [Remote host closed the connection]
WildSoft12 has joined #m-labs
rohitksingh has quit [Quit: Leaving.]
WildSoft12 has quit [Remote host closed the connection]
EspadaV8 has joined #m-labs
EspadaV8 has quit [Remote host closed the connection]
joltman1 has joined #m-labs
joltman1 has quit [Remote host closed the connection]
SupaHam4 has joined #m-labs
SupaHam4 has quit [Remote host closed the connection]
preludedrew_ has joined #m-labs
int0x27h12 has joined #m-labs
int0x27h12 has quit [Killed (Sigyn (Spam is off topic on freenode.))]
preludedrew_ has quit [Remote host closed the connection]
dostoyevsky23 has joined #m-labs
dostoyevsky23 has quit [Remote host closed the connection]
X-Scale has quit [Ping timeout: 252 seconds]
X-Scale has joined #m-labs
earlybird3 has joined #m-labs
rohitksingh has joined #m-labs
earlybird3 has quit [Remote host closed the connection]
ne4rd21 has joined #m-labs
ne4rd21 has quit [Remote host closed the connection]
rohitksingh has quit [Quit: Leaving.]
<hartytp> larsc: that wasn't the issue
<hartytp> looking at the code, there was some configuration earlier on that implemented the sequence in that table
<hartytp> well, close to it
<hartytp> I tweaked this to follow the data sheet process exactly
<hartytp> (including using separate register writes to clear the sticky bits and then re-arm the sync machine)
<hartytp> but no difference, still this odd periodic behaviour
QualityPear14 has joined #m-labs
mauz555 has joined #m-labs
mauz555 has quit [Remote host closed the connection]
mauz555 has joined #m-labs
QualityPear14 has quit [Ping timeout: 252 seconds]
<hartytp> okay, I do think this is something to do with the way we're validating the SYSREF
<hartytp> if I use one-shot then monitor mode instead then I see completely different results
bhaak6 has joined #m-labs
bhaak6 has quit [Read error: Connection reset by peer]
doskoi17 has joined #m-labs
mauz555 has quit []
<hartytp> larsc: what do you think the "recommended" way of determining the correct delay required between the SYSREF and DAC clock lines is? i.e. if there is an unknown initial delay between the two, but we can scan a delay line in the SYSREF path, how do we determine the correct delay to program?
doskoi17 has quit [Remote host closed the connection]
<hartytp> larsc: and, do you understand how to use one-shot then monitor mode? what do LASTERROR_x mean?
pfsmorigo0 has joined #m-labs
<hartytp> sbo: that's about all I can think to do for now
<hartytp> anyway, I don't think any of these issues are to do with my sync logic, I think they're to do with us not using the AD9154 sync validation logic correctly, but I don't have time to look into that
<hartytp> can you have a go with the hmc7043, please?
pfsmorigo0 has quit [Remote host closed the connection]
AntumDeluge18 has joined #m-labs
frinnst4 has joined #m-labs
AntumDeluge18 has quit [Remote host closed the connection]
frinnst4 has quit [Remote host closed the connection]
Dreg4 has joined #m-labs
Dreg4 has quit [Remote host closed the connection]
a1cypher has joined #m-labs
a1cypher has quit [Remote host closed the connection]
culot has joined #m-labs
culot has quit [Ping timeout: 268 seconds]