Uh, the bummer is when there is a metadata change, but I don't know. Um, and many times for many use cases, those trend towards zero. That's just a really, really fun thing. As the broker defaults are, which is huge because anyone who is a Kafka streams affects you. And then, uh. This means that there are no networking changes, client restarts, or offset translations to worry about, and no one needs to write any custom code to make it work. Here's why you're up at this. Right. As long as the number of replicas in the ISR is at or above the min.isr value for that partition, the partition is considered healthy and the durability requirement is satisfied. Right. So for people out there who are used to having to go in and define internal topics and Kafka streams, you can just say negative one and your replica's will be placed across a stretch cluster exactly. the Multi-Region Clusters configuration, then perform a rolling restart of the two Zookeeper instances on EAST DC. If your recovery point objective is zero means you can't lose any data than the requirements. Uh, and that's, we wouldn't want clients to have to change cause that's, you know, fairly proven stuff and you know, there's already a, you know, various streams of work in Apache Kafka or those things get improved over time and we kind of want to rely on the existing mechanism there. No, that, that makes sense. They can stay in New York, they can consume from an Insync replica and it kind of sells a lot of those issues on the consumption side, um, with latency. It's here's this new thing. Right. I actually cleaned my office recently. The end state, after youve transitioned the datacenters, will be a multi-region, rack-aware cluster that uses a Worst test ever. Because then it really does matter in terms of like latency. Correct. It means you're like we are, it means you're more, you're more polite than Ethernet, right? What's your recovery time objective. It's not a leader, it's not a follower, it's an observer note. So a few things we've got, um, Mitch, you mentioned replica placement and I guess leader placement is the thing there. Okay. Ensure all zookeepers properties have the target quorum on them. Anyway, as always, I hope this podcast was helpful to you. As of the time of this writing, the long-awaited KIP-500 is under discussion on the Kafka mailing list. I think by the time this airs, there will be a link to that available online. I think the scientific name is, is Brossa Nydia pepper FIA. So the other topic that almost always comes up is I've got, you know, maybe I don't want to have a fully synchronous replica someplace and pay that that produced latency cost. What's the deal with the KIPP number Mitch gave, which I already forget in the course of this conversation, but just, just drop it on us. Absolutely. No, wait, why won't you listen? Got it. Um, Anna and Mitch, you are both, um, technical account managers at Confluent. In Kafka, the data being written to ZooKeeper is generally quite small, so there is not much concern about the throughput of data. And I do this in my cough. Now, uh, now that I think we kind of know what a stretch cluster is and, uh, Anna, you explained why I'm reading from followers is a good idea. Producers, you got nothing you have to produce to the leader. So I have to make sure I can reproduce any data that didn't make it over as a consumer I have to know where I left off. Um, but we are talking about it, uh, straight up proprietary feature, right? So good point now, how is MRC different? That's right. I am also a napper and there's no shame in that. So, and you have the flexibility to create, you know, different strategies if you need to write, just because you can use default reckless replica placement doesn't mean you have to, if you're, you're better served using a custom one. Be sure to activate it by December 31st, 2021, and use it within 90 days after activation and any unused promo value on the expiration date will be forfeited. So it's been super fun to see the way that people are using Kafka. It is. Correct. And is it a slam dunk? Yeah. To me. If, if network has, if traffic has to leave their network, uh, it's going to hurt you. Retries do their thing. Even if you, you know, use something like replicator has offset translation, you have to deal with the fact that this is async replication, which means there potentially could be a lag, which means that I'm a producer. So as long as you have, you know, Kafka, when you have a, a partition as leaders and his followers, and there's a list that says, okay, here's all of my followers that are in sync, right. Um, sometimes there's a pattern where you have application teams deployed on, you know, the easiest way to talk about it as like coast to coast. Uh, and I was like to put that upfront, uh, not so that you can stop listening because well, they're talking about some lame commercial thing and it's just going to be a big advertisement. Alright. The, um, I would like to just say since, since the last time I was on the show, um, I, my favorite thing about being a siesta is getting to see all of the use cases because we also help people before that with like, you know, application design event, streaming, how to like, you know, come into the new, beautiful, wonderful era that is, you know, as close to real time as possible just to be accurate and event streaming. So we've kind of been digging into the client side, but I want to go there a little bit more and get a sense of what's difficult without MRC, uh, about managing fail overs. And if you subscribe through Apple podcasts, be sure to leave us a review there. Um, but so I'm trying to word it specifically, like who really cares about this? So there's yeah, so there's, there's a couple components like a stretch cluster, right? Hey, you know what you get for to the end, some free Confluent Cloud use the promo code 60PDCAST. Sure. So can I erase some of my earlier negativity? Every time I ask a question, I'll just, I'll try and balance them myself, but Anna, what is MRC? Therefore, the Since you will start ZooKeeper and brokers anew, delete the data on them: You are not deleting these in EAST DC because they are your expanding cluster. And that's totally not. Preferred leader election still takes precedence. DO NOT DELETE them. A broad outline of best practices and a sequence of steps is provided for Was it? information in each of the ZooKeeper properties files must be the same. Which would be the coolest thing ever. Have the Confluent Platform package and property file ready so that you can start it up in later steps. It's, it's the same concept of, of how we traditionally do dr. There's some, some nodes of the zookeeper ensemble in the first easy availability zone and some in the other. See my Kafka summit talk. And there's going to be, I suppose, a TCP timeout followed by some sort of exception that, uh, gets thrown and, and bad things happen. In the You're not going to pay that, that latency penalty. Okay. ROLLBACK option: Delete the ZooKeeper package and the property files from the tiebreaker node. There are various ways of running Kafka across multiple regions and various reasons why it's hard to do. It's fantastic. It's a customer success technical architect. There's no, yeah, right. And, uh, it was Halloween themed episode and it was frankly amazing. Try it free today. Um, that's pretty cool. And my cluster is, is stretched across those two locations and ZooKeeper is managing life. the 8 hour green zone does not stop data flow. Right. Yup. Yeah. And that way they don't have to go over to California and lose five pounds. Yeah. If I am going over the wan to talk to, uh, a replica leader, is that basically that problem with, you know, you said you have to administer this where the, uh, where replicas are placed and that's kind of the issue, right? They can be two things that give you a semblance of resiliency, um, as opposed to running two separate clusters and having some sort of replication between them. (No rollback actions are necessary.). For now, Apache Kafka has a single implementation of the replica.selector.class, which is a rack-aware selector. When, when broker goes down, when one leader goes offline, what's my, what's my criteria for this. The Confluent Q3 22 Launch, our latest set of Confluent Cloud product launches, is live and packed full of new features to help your business innovate quickly with real-time data, We are pleased to announce the release of Confluent Platform 7.2. It's called siesta, which is the only reason why I really like it because it, it makes me laugh. But then there's other places where they're like, well, this, you know, this business case or use case these applications right there, they're running over here on the West coast. Um, so what MRC really does is it provides an easy way to set up a Kafka cluster so that it takes advantage of all the things in stretch cluster and makes it operationally easy. You know, it's not that it's not that there's a disaster. When writing data to a partition in Apache Kafka, the preferred configuration is to set the acks producer configuration property to all. I can imagine, you know, there's latency there, but is, is the problem in the brokers is the problem with the clients. Um, and if I'm wrong, then, Hey, you're lucky. You won't be traversing the land, like when link, um, Mitch. Well, no, it's a live stream, right? Thank you for that. Where the additional ZooKeeper latency does come into play is for cluster operations like creating and deleting topics, leader election, reassigning partitions, or joining a consumer group. Um, and so it's, it's kind of a similar example, like you said, and as long as you configure your clients to be resilient in the face of retriable errors of which that is one that we'll try again until they do have a good leader elected. Hashtag bro. I feel like we're, I have a pretty good understanding of, of how this works and what the experience is like operationally and as a client developer, what are other kind of, uh, common topics that come up in the MRC world? So the way that zookeeper works and the way that, uh, kind of what defines a stretch cluster is that it's, it's one Kafka cluster with really inconvenient physical locations. It's not, it's not clear immediately to me that there would be, but do I, do I have to keep track of offsets when I manage a fail over and what, what else in my life as a developer using this gets better. Certainly, a Kafka cluster spanning multiple datacenters will have significantly higher costs for network traffic between datacenters than within a given datacenter. I just picked it out of the air. Yeah. So in that case, they'll centralize all their leaders in one DC. But, uh, remind, uh, quick Mitch quick summary. You don't, you don't want to have to do that. Yup. Those are replicated asynchronously. This class was intentionally made into a pluggable interfaces so users can supply their own implementation depending on their needs. And I gave you, and I gave the Kip number, just a palindrome. for all 3 ZooKeepers in each of the zookeeper.properties files, as shown: Zookeepers will not join if they do not know each other. I'll just say data centers for short. Right. Right. It also, you know, side benefit a little bit, maybe, you know, spreads the load on whatever node is currently the leader for that petition a little bit. So, um, I think you're gonna find the conversation interesting. Yes, it does. And to this podcast, wherever fine podcasts are sold. Um, and I have, I have, uh, you know, good old fashioned socket connections to broke between the client and the broker. She had one cluster, sorry. And so, you know, those should really drive your requirements without knowing those. There's all this stuff going on inside the client, right? Confluent Cloud is a fully-managed Apache Kafka service available on all three major clouds. He also authored a popular Python client for Kafka which received wide adoption, although he now recommends Confluents client . Check the ZooKeeper logs to make sure everything is running properly. Hello, and welcome back to another episode of Streaming Audio. What kind of plant is it? However, this architecture was only viable if data centers were very close to one another because of the cost to throughput, the operational overhead of dealing with replica placement, and the volume of cross-DC traffic the architecture creates. and replica placement available in Multi-Region Clusters. Cause it seems self evident to me that you'd want disaster recovery. Replicator for Multi-Datacenter Replication. This functionality allows operators to increase data durability and automate client failover in the event of a disaster. This is a Confluent Platform feature only, and is not available on Confluent Cloud. And then your recovery point objective is how much of this live stream can I afford to miss like 10 minutes, you know, 20 none do I have to pick up right where I left off. Where do you need to wrap the cause? within your Confluent Platform install directory. Now we're still talking about stress clusters. But in the end of the day, that tends to be less reliable than your local data center way land, uh, on the operation side for Kafka. Is it? Replicator is no longer needed in for a Multi-Region Clusters setup. You have to do catching an exception or anything like that. It's just, you want to be available. On this episode of Streaming Audio, a podcast about Kafka, Confluent, and the cloud. It just has latency characteristics. consumers, producers, and stopping replication. Hashtag one cluster use it, love it. Uh, no it's actually defined. Previously, any replica not belonging to the ISR was considered out of sync. And it's very obvious, usually not right there. Watch me. Yeah. For the, for the teams out there with a, you know, a multitenant cluster, they don't have to go to each application team as they onboard and say, what are you trying to do? So what you end up doing, you say, okay, I'm going to place my replica, such that I know there's availability in the New York, DC for these consumers to fetch from a follower. You know, when you begin the discussion, you're struggling to understand what their use cases and wondering, wow, do I even have answers for this person? Yeah. My, you know, replicas all of these things in order to ensure resiliency and Mitch, um, like right from the gate has done a ton of work with us. It's not just Kafka streams. Natto, obviously you love it. We would produce the message into one cluster and it would eventually get out to the other cluster. This time we have focused on improving your experience with aggregate functions, JOINs, and schemas. Essentially, this is how Apache Kafka provides durability. And it's all good. Um, you know, since my time here now. Confluent Platform 5.4 changes all of that. Yeah. I am your host, Tim Berglund and I am, I think, particularly delighted today to be joined in the virtual studio by two guests Anna McDonald and Mitch Henderson. I want to make sure that the, whatever, you know, whatever downsides, I guess I want to lean into niches, negativity, whatever downsides there are of stretch clusters. Okay. Youll begin by freeing up resources on the passive cluster; pausing all I want to make sure they're on the table. So as long as you're contributing properly. And that's where this architecture really, really shines because it is just one cluster, all the things that those clients do, all that plumbing you're talking about, we just inherit that. ROLLBACK option: Restart this East ZooKeeper with the previous properties. Right. What do you do? All other trademarks, servicemarks, and copyrights are the property of their respective owners. In order to complement the new Observers feature, and to further enable practical multi-datacenter deployments, we have created a new replica placement strategy for Confluent Platform. Uh, ball caps, stickers, um, hoodies. Anna. That feature is less than six months old. So I think maybe I'm just a little awestruck, yeah. And I always liked that because if you think about it, right, there are times where you have a batch job and it fails and you don't care.