I'm kind of surprised no one has tried doing this before...or at least I haven't found anyone attempting to do this. Perhaps it's too closely related to SSH for people to bother. Basically, what I am trying to do is create a class (in Java) that launches a subprocess, and in which the stdout, stderr and stdin of the process are all linked to a network socket. The key advantage to this over SSH is that unlike SSH, if you close your terminal, you don't also kill your process(es) (though I'm not sure if there's already a way to do this in SSH). And of course, this will be applicable to windows platform.
I'm using the Netty project currently as my low level NIO network framework. It's not too bad, but I do seem to be having some trouble getting the asynchronous event driven nature of Netty working with the thread reading mechanic of the stdout and stdin reader of the subprocess. Basically, in order to read the stdout of the subprocess, I have a separate thread running which is constantly checking to see if there's anything in the process's InputStream (yeah, confusing...go look at the java.lang.Process class, but stdout is actually an InputStream).
So what I have to do is somehow, in the Runnable's run() method, pass Netty's Channel object to the Thread object and write the output to the channel. I can currently see when a client connects to the server, and the server can send a basic message, but I can't seem to pass the (shared) channel object (which I assign when the channel is connected) and have the stdout reader thread be able to use it.
Still, I feel like I'm on the right track. And since I think this could be a useful project, I'll open source this. Eventually, if I ever get it working, I'll port it to clojure. I'd eventually like to add a security layer (Netty has SSL support). But first things first.
Wednesday, June 29, 2011
Thursday, June 16, 2011
Functional python
Sadly, I don't have an opportunity to write clojure at work, but I am able to write in python, so I've been tackling some common problems in python in a more functional style. I've discovered that list comprehension, lambdas, map, and reduce are your friends. Also, writing in a functional style often means writing less iterative code, and more recursive code.
So first things first, how can list comprehensions help? Imagine if you a list of characters, and you want to combine all of them into a word. Of course the pythonic way to do this would be:
example = [ 'h', 'e', 'l', 'l', 'o', ' ', 'w', 'o', 'r', 'l', 'd']
"".join(example)
This is of course perfectly valid. A functional approach to this would be this:
reduce(lambda x,y: x + y, example)
In this case, the lambda is adding (concatenating) two strings together. The reduce function takes the first two arguments, and then applies the result of that to the next argument.
Using the map() function is also handy, and can sometimes be easier to read then list comprehensions. For example, the two below are equivalent:
[ x**2 for x in (1,2,3,4) ]
map(lambda x: x**2, (1,2,3,4))
However, what if you wanted to multiply some_collection[x] + another_collection[x]? If you try to do this as a list comprehension, you won't get what you think:
[ x * y for x in (1,2,3,4) for y in (10,20,30,40) ] ## try it
Instead, you can use a map here:
map(lambda x,y: x*y, [1,2,3,4], [10,20,30,40])
While the examples above have been relatively trivial, here's a more complicated scenario. Imagine that you are given an amount of change, and you are to calculate all the possible combinations of quarters, dimes, nickels and pennies you can get. For example, if you are given 27 cents, you could have:
1 quarter, 2 pennies
2 dimes, 1 nickel, 2 pennies
1 dime, 2 nickels, 2 pennies
etc etc.
So how would you go about doing this? When I first thought about this problem, I tackled it in the usual manner by trying to come up with an iterative imperative solution. But I later decided (after the problem being fresh out of my mind) to come up with a recursive solution. However, it's not only recursive, it's a mutual recursive problem.
So think about the problem like this.
1. I have a total. Given a number of quarters Q, I know the remainder of change (total - (25 * Q) )
2. I have a remainder. Given a number of dimes D, I know the remainder of change ( remainder - (10 * D))
3. I have a remainder. Given a number of nickels N, I know the remainder of change ( remainder - (5 * N))
4. Any remainder left must be pennies
Do you see how all the problems are similar? there are some gotchas however. But below is the code representing my solution to this tricky problem. I used a list here as return values so I could add lists together when one function call popped off the stack.
import pprint
def remainderNickels(total, nickels):
if (nickels * 5) < total:
newn = nickels + 1
return remainderNickels(total, newn)
else:
return [ {'pennies' : total - (5 * (nickels - 1)), 'nickels' : nickels - 1 }]
def remainderDimes(total, dimes):
remainder = total - (dimes*10)
if (dimes * 10) < total:
newd = dimes + 1
if remainder >= 5:
return remainderDimes(total, newd) + [ { 'remainder' : remainderNickels(remainder, 1), 'dimes' : dimes }]
else:
return [ {'pennies' : total - (10 * dimes), 'dimes' : dimes }]
else:
return [[]]
def remainderQuarters(total, quarters):
remainder = total - (quarters*25)
if (quarters * 25) < total:
newq = quarters + 1
if remainder >= 10:
return remainderQuarters(total, newq) + [{ 'remainder' : remainderDimes(remainder, 1), 'quarters' : quarters} ]
elif remainder >= 5:
return remainderQuarters(total, newq) + [{ 'remainder' : remainderNickels(remainder, 1), 'quarters' : quarters} ]
else:
print "remainder = ", remainder
return [ {'pennies' : remainder, 'quarters' : quarters }]
else:
return []
q = remainderQuarters(88, 1)
pp =pprint.PrettyPrinter()
pp.pprint(q)
So first things first, how can list comprehensions help? Imagine if you a list of characters, and you want to combine all of them into a word. Of course the pythonic way to do this would be:
example = [ 'h', 'e', 'l', 'l', 'o', ' ', 'w', 'o', 'r', 'l', 'd']
"".join(example)
This is of course perfectly valid. A functional approach to this would be this:
reduce(lambda x,y: x + y, example)
In this case, the lambda is adding (concatenating) two strings together. The reduce function takes the first two arguments, and then applies the result of that to the next argument.
Using the map() function is also handy, and can sometimes be easier to read then list comprehensions. For example, the two below are equivalent:
[ x**2 for x in (1,2,3,4) ]
map(lambda x: x**2, (1,2,3,4))
However, what if you wanted to multiply some_collection[x] + another_collection[x]? If you try to do this as a list comprehension, you won't get what you think:
[ x * y for x in (1,2,3,4) for y in (10,20,30,40) ] ## try it
Instead, you can use a map here:
map(lambda x,y: x*y, [1,2,3,4], [10,20,30,40])
While the examples above have been relatively trivial, here's a more complicated scenario. Imagine that you are given an amount of change, and you are to calculate all the possible combinations of quarters, dimes, nickels and pennies you can get. For example, if you are given 27 cents, you could have:
1 quarter, 2 pennies
2 dimes, 1 nickel, 2 pennies
1 dime, 2 nickels, 2 pennies
etc etc.
So how would you go about doing this? When I first thought about this problem, I tackled it in the usual manner by trying to come up with an iterative imperative solution. But I later decided (after the problem being fresh out of my mind) to come up with a recursive solution. However, it's not only recursive, it's a mutual recursive problem.
So think about the problem like this.
1. I have a total. Given a number of quarters Q, I know the remainder of change (total - (25 * Q) )
2. I have a remainder. Given a number of dimes D, I know the remainder of change ( remainder - (10 * D))
3. I have a remainder. Given a number of nickels N, I know the remainder of change ( remainder - (5 * N))
4. Any remainder left must be pennies
Do you see how all the problems are similar? there are some gotchas however. But below is the code representing my solution to this tricky problem. I used a list here as return values so I could add lists together when one function call popped off the stack.
import pprint
def remainderNickels(total, nickels):
if (nickels * 5) < total:
newn = nickels + 1
return remainderNickels(total, newn)
else:
return [ {'pennies' : total - (5 * (nickels - 1)), 'nickels' : nickels - 1 }]
def remainderDimes(total, dimes):
remainder = total - (dimes*10)
if (dimes * 10) < total:
newd = dimes + 1
if remainder >= 5:
return remainderDimes(total, newd) + [ { 'remainder' : remainderNickels(remainder, 1), 'dimes' : dimes }]
else:
return [ {'pennies' : total - (10 * dimes), 'dimes' : dimes }]
else:
return [[]]
def remainderQuarters(total, quarters):
remainder = total - (quarters*25)
if (quarters * 25) < total:
newq = quarters + 1
if remainder >= 10:
return remainderQuarters(total, newq) + [{ 'remainder' : remainderDimes(remainder, 1), 'quarters' : quarters} ]
elif remainder >= 5:
return remainderQuarters(total, newq) + [{ 'remainder' : remainderNickels(remainder, 1), 'quarters' : quarters} ]
else:
print "remainder = ", remainder
return [ {'pennies' : remainder, 'quarters' : quarters }]
else:
return []
q = remainderQuarters(88, 1)
pp =pprint.PrettyPrinter()
pp.pprint(q)
Monday, June 13, 2011
Things I need to get better at
Being a jack-of-all-trades suits me. I enjoy being able to dabble in many diverse areas of technology. Although my knowledge is shallow in many areas it is broad. I am a decent programmer in many languages, and I have in my short career worked with embedded device drivers and firmware, writing proprietary messaging protocols over TCP/IP sockets, stored and queried needed information on both MySQL and H2 databases, wrote XMLRPC servers, RMI servers, written JNI wrappers around C shared libraries, and many things inbetween. In other words, I have a pretty good view of the entire technology stack.
But there are still many areas I need improving on. Now that I am writing more java again, and I wish to be better at clojure, I need to get better at the java ecosystem. For example:
1. Get better at maven. I have built a few non-trivial maven projects, including one multi-module, but a lot of maven's finer points still elude me.
2. Javadoc commenting. I just let Eclipse auto-fill in the params and return values, but I really should know all the markup for it.
3. Annotations. I understand them in theory, but I've never written one (same goes for python decorators)
4. Unit testing. Yeah yeah, being in Test, I should know JUnit or TestNG like the back of my hand. While I realize their importance, sadly, time constraints often win. I have tried writing some TestNG unit tests, but they are not being called from maven, and I haven't had time to figure out why
5. OSGi. While I have written two eclipse plugins, I still don't truly understand a lot of OSGi. I understand what it's for (modularity to decrease coupling, and provide metadata to end jar hell), but it's such a huge beast that I need to know more
There's also a lot in general that I want to get better at or relearn:
1. C/C++. I haven't seriously written any C or C++ in a little over a year when I was doing some JNI wrapping. The new features in C++0x looks interesting, and eventually, I hope to get back to more JNI programming.
2. Advanced python. By this, I mean stuff like decorators, generators, continuations, and metaprogramming. I actually think I finally understand generators (functions that yield an iterator like object), and how they can be used for continuation style programming. I once showed a coworker how to implement "private" methods and fields in python through implementing some of the magic methods, but that's the most metaprogramming I've done.
3. Algorithm design. After implementing a homegrown software dependency installer program, I came up on my own a depth first search algorithm that could do post-order traversal. I didn't know it was called that until after I read the chapter on traversal algorithms in the book Python Algorithms: Mastering Basic Algorithms in the Python Language.
4. Concurrent programming: Most of my experience with multi-threading has simply been to spawn a new thread to prevent blocking during a long running task. Only twice have I had to share data across threads, and honestly, I'm not sure if I synchronized things right. One reason I wanted to learn clojure was for its approach to concurrent programming (even OpenGL is massively parallel in nature).
And while all of the above will help me in the "real" world, for my own personal desire, I still want to learn OpenGL, and get better at clojure. I also want to get better at JBoss's Netty NIO framework. Sadly, much of my spare time is spent working for work.
But there are still many areas I need improving on. Now that I am writing more java again, and I wish to be better at clojure, I need to get better at the java ecosystem. For example:
1. Get better at maven. I have built a few non-trivial maven projects, including one multi-module, but a lot of maven's finer points still elude me.
2. Javadoc commenting. I just let Eclipse auto-fill in the params and return values, but I really should know all the markup for it.
3. Annotations. I understand them in theory, but I've never written one (same goes for python decorators)
4. Unit testing. Yeah yeah, being in Test, I should know JUnit or TestNG like the back of my hand. While I realize their importance, sadly, time constraints often win. I have tried writing some TestNG unit tests, but they are not being called from maven, and I haven't had time to figure out why
5. OSGi. While I have written two eclipse plugins, I still don't truly understand a lot of OSGi. I understand what it's for (modularity to decrease coupling, and provide metadata to end jar hell), but it's such a huge beast that I need to know more
There's also a lot in general that I want to get better at or relearn:
1. C/C++. I haven't seriously written any C or C++ in a little over a year when I was doing some JNI wrapping. The new features in C++0x looks interesting, and eventually, I hope to get back to more JNI programming.
2. Advanced python. By this, I mean stuff like decorators, generators, continuations, and metaprogramming. I actually think I finally understand generators (functions that yield an iterator like object), and how they can be used for continuation style programming. I once showed a coworker how to implement "private" methods and fields in python through implementing some of the magic methods, but that's the most metaprogramming I've done.
3. Algorithm design. After implementing a homegrown software dependency installer program, I came up on my own a depth first search algorithm that could do post-order traversal. I didn't know it was called that until after I read the chapter on traversal algorithms in the book Python Algorithms: Mastering Basic Algorithms in the Python Language.
4. Concurrent programming: Most of my experience with multi-threading has simply been to spawn a new thread to prevent blocking during a long running task. Only twice have I had to share data across threads, and honestly, I'm not sure if I synchronized things right. One reason I wanted to learn clojure was for its approach to concurrent programming (even OpenGL is massively parallel in nature).
And while all of the above will help me in the "real" world, for my own personal desire, I still want to learn OpenGL, and get better at clojure. I also want to get better at JBoss's Netty NIO framework. Sadly, much of my spare time is spent working for work.
Subscribe to:
Posts (Atom)