I didn't realize it's been so long since I've written a blog. I am going to try and correct that and write at least 2 blogs a month. I've been up to a few things, so I'd like to talk about them as things progress.
The first thing though is that I've been at work on a messaging system I've named pheidippides. It's partially a learning project, and partially something I can use at work. In essence, it's a system and protocol that will handle both RPC and pub/sub style messaging. Both of these will be handled in an asynchronous and decoupled nature. Pheidippides is all about sending messages from one Module to another.
Encoding:
Rather than use JSON or XML to represent a message, it will use the binary encoded msgpack
Protocol:
Instead of using http with REST style mechanics, pheidippides will have its own protocol for interpreting messages
Transport:
While the first focus of pheidippides will be on TCP/IP, it will be abstracted enough to handle native (in-memory) transport, and some investigation into supporting dbus is also being done
Why yet another messaging framework?
Partially as I stated above, pheidippides is a learning project. I've been curious about messaging frameworks and all the hoopla surrouding them. From a more pragmatic standpoint, I wanted to replace our test framework's rampant usage of making SSH calls to do things. I've also been considering making something akin to Ansible or Puppet, but using pheidippides as the means to "get stuff done". Also, at work, we heavily make use of dbus messages, and I think I can make an adapter from pheidippides messages to dbus messages (given enough time). If I can do this, it would be a huge advantage by being able to create a bridge between dbus and pheidippides (I'm gonna call it pdd for short from now on...reminder to self...make a shorter name for projects). By having this bridge, any Module hooked into pdd would also get dbus events. I'm also not unfamiliar with low-level protocols. At my first job, we basically had a router who had attached services, and at LSI, we had message passing system between the (user library) driver and the firmware.
Also, most other messaging systems fall short for one reason or another.
HTTP w/REST: Is still a request/response (synchronous) way of doing things. Also, there's a lot of overhead with JSON or XML not to mention all the baggage of http
Websockets: Websockets solves the synchronous problem of REST, but then you're left with implementing your own protocol anyway
AMQP: Comes close, but it's RPC mechanism takes a little work, and at least as I understand it, it requires a coupling of caller and callee
WAMP-proto: I actually just discovered this yesterday, and it comes the closest to what I have been envisioning. However, it doesn't use a binary encoding of messages, and although they say they will support outside of websockets, they appear to not have done so yet.
There's a ton of work I still need to do. Here's a rough overview I've made
Right now, pheidippides isn't actually useable. I just recently got it working to send a simple register message, but the Controller isn't even doing anything with it yet. But, at least I got the encoding/decoding and transport for TCP/IP working. If you look at the issues list, there's still a ton I need to do.
But I have to say, I'm having quite a bit of fun with this. I'm learning a lot about networking on Java, as well as NIO2. I have an idea to write a macro for something which I feel is a shortcoming in clojure (namely, that defrecords can't inherit...if you have defrecord A with fields foo, bar and baz, and I want another defrecord B that has those same fields but additionally quux, there's no clean way to do it in clojure without repeating the fields).
I'm also stoked to get pheidippides up and running to replace all of our SSH calls in our test code. While SSH calls work ok, for one, they are very non-JVM'ish. Basically we're coding in bash or shelling out to a subprocess, and then scraping the results. That's just fugly in my opinion. In fact, our code doesn't even get SSH output (or ProcessBuilder output) in real time (but my other library commando fixes that...although I've not replaced it in our testware yet). Eventually, I'd like pheidippides to be the foundation for an agent based system to provision our test VM's. The hotness seems to be Ansible, but while Ansible touts as an advantage that it doesn't require an Agent, I also view that as a weakness (because Ansible is a push-only system, it can not get events or messages that are not explicitly called for).
But beyond work, I envision using pheidippides as the basis for a real-time distributed analysis system. I'm even going to look into Infinispan for the database, as that would also give me a distributed data-grid and distributed execution.
Finally, I'm going to move most of my blog posts to rarebreed.github.io (once I figure out how to use jekyll). I like the fact that blogs are just markdown files, and that would let me share code examples and snippets much more easily.
But I have to say, I'm having quite a bit of fun with this. I'm learning a lot about networking on Java, as well as NIO2. I have an idea to write a macro for something which I feel is a shortcoming in clojure (namely, that defrecords can't inherit...if you have defrecord A with fields foo, bar and baz, and I want another defrecord B that has those same fields but additionally quux, there's no clean way to do it in clojure without repeating the fields).
I'm also stoked to get pheidippides up and running to replace all of our SSH calls in our test code. While SSH calls work ok, for one, they are very non-JVM'ish. Basically we're coding in bash or shelling out to a subprocess, and then scraping the results. That's just fugly in my opinion. In fact, our code doesn't even get SSH output (or ProcessBuilder output) in real time (but my other library commando fixes that...although I've not replaced it in our testware yet). Eventually, I'd like pheidippides to be the foundation for an agent based system to provision our test VM's. The hotness seems to be Ansible, but while Ansible touts as an advantage that it doesn't require an Agent, I also view that as a weakness (because Ansible is a push-only system, it can not get events or messages that are not explicitly called for).
But beyond work, I envision using pheidippides as the basis for a real-time distributed analysis system. I'm even going to look into Infinispan for the database, as that would also give me a distributed data-grid and distributed execution.
Finally, I'm going to move most of my blog posts to rarebreed.github.io (once I figure out how to use jekyll). I like the fact that blogs are just markdown files, and that would let me share code examples and snippets much more easily.
