Serializing & Deserializing JSON

[00:00:00]
>> Josh Long: Okay, so configuration. What do we got to do? Well, we've got the group ID already here. I'm not sure if we need to specify it, but we can. We want to specify the deserializer. So Spring Kafka deserializer, right. So when you get messages for the key. Right, the key we talked about, that has to be deserialized and the value has to be serialized.

[00:00:17]
So I'm going to specify a generic Kafka deserializer. That'll do, I think. JSON or maybe it's bytes or something. It's pretty smart. But the point is you have to like JSON. Let's just do JSON. Okay, so there's the key deserializer, and here is the value deserializer. Okay, simple.

[00:00:37]
That was pretty simple and then what about the producer? Remember, we're not just reading values from Kafka, we're also writing values to Kafka. So the producer has to also serialize the data. So I'm giving it just a regular Java object somewhere between once I've handed it a Java object and it reaching the network wire protocol for Kafka.

[00:00:55]
Somewhere along the lines that value needs to be serialized. So we say producer serializer, okay. Key serializer and same thing, right, JSON. I'm going to use the JSON serializer instead of deserializer, okay. Okay and then same thing over here for the value, okay. Looks fine, I guess. I don't know.

[00:01:21]
We'll try it out. Let's see if that works. Having started up a brand new Kafka in my Docker image, I don't have the web, okay. Since I'm using JSON, it needs the JSON marshalling, which in turn implies that I need the web stack, which is a bit unfortunate.

[00:01:42]
So I added spring boot start a web to my build. That'll give me the JSON support, which is a function of that. So there we go. My group partitions assigned. Did we get a message? When the Kafka driver starts up, you get all this very verbose output. So it's very hard to know.

[00:02:09]
Did it work? Did it connect? Do I need to specify the same consumer group? You might have to specify the consumer group. Yeah, consumer client id, Group id. No, group ID is my group. Okay, try that over here. Try that there. Restart this. Okay, now we got the.

[00:02:38]
Okay, we got something, so stop it. So now we got the next error, which is that this type is not in the trusted packages. This is another thing. So remember we are receiving a JSON payload over the Wire and the payload has information about the class that we're trying to deserialize it to.

[00:02:57]
Would you trust random unscrutinized packets over the network to serialize into Java types? Right to actually result in Java types being created. So you need to actually allow Kafka to do that. I think there's a property, let me see. So consumer properties. Okay, and then there's this magic string.

[00:03:19]
JSON.trusted packages equals *. Okay, so what that's saying is I want to implicitly trust every message coming in. Don't do that. You can also give it a commons, a limited list to be very specific. But for now, for this demo to succeed, make sure you add that property.

[00:03:35]
Okay, properties spring dot. There you go. There we go. Okay, so once I've done that, all the three messages that were delivered before that errantly didn't get delivered, they finally got delivered retroactively. So I was able to, with a fair bit of configuration, actually there's more configuration arguably than there's actual Java code.

[00:03:58]
The producer is this bit. I'm injecting the Kafka template and I'm calling kafkatemplate send, sending it to a well known topic. On the consumer side, I have a Kafka listener. This could be two different JVMs and as long as they have the same host and port for the Kafka broker, then they will work.

[00:04:14]
On the consumer side, I need to make sure I trust all packages for deserialization. I need to specify and care about what serializers are used for production and consumption. But then at the end of it, I have a working service, so I'm not even sure if I needed this.

[00:04:26]
Actually, let's get rid of that, try it one more time and see if we can do without that. Yep, I could have. So there you go. Just specify the producer and the consumer serializers for the key and the value, specify that you want to support all JSON types and then you're done.

[00:04:42]
Okay, that's Kafka. And if you want to do pulsar while you add that to the class path, specify the equ equivalent configuration and you do this, right? If I want to do rabbit rabbitmq, you do this. You get the idea, right? It's as easy as it can be.