Using Hyperlog

[00:00:00]
>> So a library that I like to use in OJS, it sits on top of leveldB. To do this kind of stuff, is called hyperlog. So hyperlog gives you merkle DAGs by hashing the content of every message. And when you wanna refer back to previous messages, you use the hash.

[00:00:16]
And it also has some hooks, so that you can add things like signing by default. So, it doesn't do signing itself by default, but you can add the signing pretty easily, and I've got a cool package that I can show you how to use that with. So here's a really simple example of how to create a log with hyperlog.

[00:00:39]
It's just like we've been doing you require level. You instantiate your level instance with a directory, and you can. Give that to hyperlog, and then hyper log also takes a very similar key, very similar value encoding properties. So if you want to store JSON objects, you can do that by passing value encoding JSON into hyperlog or by default, it lets you store binary buffers.

[00:01:10]
So why don't we go ahead and make a little data feed with hyperlog. So, first thing we can do is we'll write a little program that will add a message, and maybe it'll print out what the latest hash is so that we can sort of fold those hashes like cascade them from the current document and to the next one, to the next one.

[00:01:36]
Okay, so require level like we've been doing all day, over and over again. Hopefully I've drilled it in by now. Make a directory to put the stuff like log.dB. If you wanna use the browser, you could just require level cache browserify. And all of this would work just the same, but here we're just gonna be sticking to node for a little bit.

[00:02:01]
Now we can require the hyperlog package, you have to npm install it first. So once you've done that, then you can make a new hyperlog instance. So we're gonna hyperlog takes the database as first argument, and optionally you can provide some other configuration like value encoding. Which I will do, because it's pretty handy.

[00:02:24]
Okay, so now with our program, we can add messages to our log. So what that looks like, just like we were doing with our merkle DAG example on the command line with Shazam. But, now this library is gonna handle a lot of that stuff for us. So, hyperlog has a method called add, the arguments to add, the first one is an array of the links that we've seen previously.

[00:02:51]
So I'm gonna compose that from the process that argv.slice2. And what this does is, all of the things that, actually that's slice3, because we need a message as our first argument. Okay, so how this program is gonna work, is we're gonna pass in as the first argument, a message we wanna save.

[00:03:14]
And then we're gonna pass in whatever links we want to be considered our predecessors of the current document. So we pass in the links, and then we can pass in our document, which because I set value encoding to be JSON, we can make it be arbitrary JSON. So, I'll just call a message msg, and also save a time standpoint date.now.

[00:03:44]
Of course we can't trust that timestamp, my computer could be lying, who knows. Not like we can trust the cryptographic properties of merkle DAGs, which are extremely reliable, so long as those hash functions are good ones. Anyways, the callback that we provide to the add function, it's gonna get some values like we're gonna get the node object that we inserted.

[00:04:13]
So I'm just gonna print out what that is because it'll have useful information like the key, that should be the hash of the document that this was stored as. There's an error, we wanna know about it. Otherwise, we'll print the node. So, if everything is working well. The first time we run this program, we won't pass in any links because we don't have any predecessors to point to.

[00:04:35]
So I'm just gonna run it with this message, hello. And here, I get back to subject. The object has some stuff in it, this sort of internal type or log, but here's the key and that key is the hash of our document and also of the links that we provided, because it's important that the links are embedded in the thing that's going to be hashed.

[00:04:59]
And here's our value. The hyperlog API is also pretty nice. It's kind of similar to leveldB in a lot of ways, it has a batch function where you can insert multiple documents atomically. It also has a create read stream function that works in a similar way to leveldB.

[00:05:17]
It also has a replicate function that just lets you do peer to peer replication very easily with duplex streams. So we're gonna be messing with those things in a moment. But for now, I'm just gonna add a couple more documents and then we can list out everything from the database.

[00:05:33]
Okay, so what are some other messages. So the next time we make a message, like this is the second one, we need to link back to the first document. And we can do that by copying this key, and putting it into the links array. So we call log.add links, now includes this hash from the predecessor and the document.

[00:06:01]
So I do that. So now in this message, before there weren't any links, and now we get this link to the first document. So we've kind of have this chain of custody going. I'm gonna put a few more documents in here. Like this is the third one, maybe third one, just to mix it up a little bit.

[00:06:24]
And now I'll copy this key, paste it in. Make some more, 444. Copy that in. Okay, cool. So now we've got some documents stored in our merkle DAG in leveldB. So let's make a program now that's just can list out everything there. So, call this one list.js. And now instead of doing all of this stuff, we can just do log createReadStream.

[00:06:57]
And we can pipe that into to, to2, sounds really silly when I say it out loud, but that's what it's called. So, just like I've been doing all day. We can just print out all of the records. Maybe just like the value as well. Well, maybe we wanna see the whole thing, I don't know.

[00:07:22]
So whenever in that program, now we see all of the documents from the history. And you can see that they link back to each other. So.