00:00 <SeanTAllen> but if the sender were responsible to keeping a queue and we could say "eveything up to Foo has been handled..." that seems reasonable

00:00 <SeanTAllen> and would include a reconnect handshake to say that "we last full processed message with id 500 from you"

00:02 <nemosupremo> I feel like the TCPSource/Sink I should do whatever I want, it feels very raw. What I was thinking is I would carry the offset id all the way to the end of the pipeline (to the encoder), and encode the offset id back to me sink

00:02 <nemosupremo> Then I would decide how often I would persist that recovery information

00:02 <nemosupremo> but I guess that could clash with wallaroo's internal state management

00:03 <SeanTAllen> it would indeed

00:03 <SeanTAllen> but what we have that wouldnt is we track your message to completion and inform you of your id when its done

00:04 <SeanTAllen> but we do that on the basis of something like this

00:04 <SeanTAllen> we get message 1,2,3,4,5

00:04 <SeanTAllen> when 1,2,3,4 are done, we ack back to you that 4 is done so you know that up to 4 is done

00:04 <SeanTAllen> that we can do

00:05 <SeanTAllen> and if you wanted to track your id through, you could make it part of the message data

00:07 <SeanTAllen> but having a different source that is TCP that exposes a more complete and raw lifecycle is an interesting idea that i hadnt previously thought of

00:08 <dipin> nemosupremo: Understood regarding the lack of polish on the Pony Kafka client. I'm somewhat confused about your comment regarding the kafka key not being readable (the kafka client itself doesn't hide it so it might be a wallaro source api issue instead) but I'll check the archives for your comments regarding that when I have a chance.

00:09 <nemosupremo> dipin: From what I understand this isn't an issue with the pony client, but the wallaroo API

00:10 <SeanTAllen> i need to head off for dinner soon. thanks for your feedback nemosupremo, you've given me a very interesting new idea to think about.