OpenStack clojure tool

So, I'm at Red Hat now :)  I'm now a Quality Engineer working on OpenStack which is a new direction for me.  This is the first time I have been working on a 100% software project.  Well, that's not entirely true I guess.  I did spend 18 months designing an automation framework from scratch.  Nevertheless, this is new and interesting.

That being said, OpenStack is written almost entirely in Python.  Namely Python 2.7. uggh.

Why the moaning?  I used to be a big rah-rah python guy.  A former co-worker of mine even jokingly wondered if Guido Van Rossum was paying me money to try to switch our company over to Python (which by the way, I pretty much single handedly did).  However, over the years, I have come to find many pain points with the language that has considerably dimmed my enjoyment of the language.  Now, hopefully I won't get any flames.  Python isn't a bad language and it has quite a few interesting features.  I just find myself longing for some things python lacks.  And indeed, with some really interesting dynamic interpreted (or JIT'ed) system programming languages like Go, Julia, and even Swift, I really wonder how much wind is going to get taken out of Python's sails in the next 5 years?  And that doesn't even factor in the non-system's programming languages like Clojure, Elixir or even TypedScript or LiveScript (a haskell-like variant of javascript).

Duck Typing ain't enough
I think Python 3 came to this realization with their new argument annotations, and so Python 3 doesn't suffer from this problem like Python 2 does.  Type hinting is the way to go.  It allows the developer to rapidly prototype an idea, and then for performance or documentation reasons, type the variables and return code later.  It would be nicer if Python was like Julia (or TypedClojure) and allowed even locals to be optionally typed.

When you start getting code bases into the many tens of thousands or more of code, you just look at a function and wonder "ok, what kind of variable am I supposed to pass in?".  Some of you may be saying that's what a good docstring is for.  I would agree, except that we all know the first thing to bit-rot is documentation.

Moreover, duck-typing can lead to unintended problems.  Perhaps you want to pass in an object that supports the method quack().  Unfortunately, the user so happens to pass in a BadDoctor class object, and your function happily calls the quack() method for you.

Hard to make constants (immutables)
Basically, if you want to truly make an immutable object in python, you'll need to subclass from int, tuple, string or some other immutable built in type.  And it's a little odd to do so.  It's one of the few places that implementing __new__() is required.  I often tell people that python's __init__ is not the constructor, __new__ is.  It is __new__ that actually allocates the memory for the object, and __init__ initializes the allocated memory.  If you have an immutable object, you have to give it a value as soon as it is created.

Another way to make "read-only" objects is to use a setter property.  It's not fool-proof, but it does allow one to make a mostly read-only object.  You could also reimplement __getattr__ and __setattr__ for the class and have it look up what you are trying to access.  And lastly, you could write a C(++) module for the data structure which does have const.  But really, would you want to do that?

Performance
Pypy aside, python's performance leaves something to be desired.  It also seems that Guido is totally nonplussed by python's performance and thinks it's good enough.  I was quite startled to recently learn how good the V8 Javascript engine performs.  That's not bad at all, and would make it on average about as fast as PyPy.  But Openstack requires regular CPython, mainly because of lots of dependencies on modules that use C modules (when you install OpenStack from devstack or packstack, you'll see some source compilation going on).

The Browser as the new VM
Like it or not, the browser is kind of the new VM.  That means that javascript is becoming as important as C or Java and just as ubiquitous.  Having an application that can run virtually anywhere, including mobile devices is not to be scoffed at.  Also, I was surprised to learn the new tricks HTML 5 has up its sleeve.  This includes a File System API so that you can finally read local files (albeit to a sandboxed file system), the websocket API, WebGL, and drag and drop support just to mention a few.  Since javascript is the de facto language of the browser, that means for better or worse learning javascript.  There are quite a few python-to-javascript libraries out there, including pyjs, and brython. However, they are not developed by the core python team and so I wonder if/when support will end?  And brython only supports python3.

No persistent data structures built-in
So there is pysistence.  But being a non-standard 3rd party library and with the lack of data-typing, it means that users will not be sure when persistent or non-persistent data types are being used.  Why do we want persistent data structures by default?  This page and this one sum it up pretty well.

Lack of good concurrency
Python, thanks to the GIL, doesn't really have true concurrency.  There is multiprocessing, which fires up a new python interpreter, but it does have some limitations (like the arguments must be pickle-able on Windows) which can be a real pain.  Also, since a new python interpreter is getting fired up for each new multiprocess, python developers can't really laugh at the JVM's large consumption of memory once you start firing up 20+ processes.  Hopefully pypy will solve this problem with Software Transactional Memory.

Projects to work on

So a few days ago I came up with a list of some fun things to work on to expand my knowledge and proficiency.  I'm putting the language design on the backburner just because there's so much you have to know to build your own language.  That being said, I will learn more about language design via another route.  So here's a list of some things I could work on:

• Data structures: Trie, Persistent RB Tree, B+ Tree (julia, clojure).
• Build a simple filesystem (julia, clojure, rust?)
• Convert Machine Leairning code (julia, clojure)
• Build a graph database (jula, clojure)
• Build messaging framework (julia, python)
• Build a parser combinator (julia, clojure)

If you'll notice, all these projects are what you would find in a typical computer science class.  As I've explained in other blog posts, one's foundation has to be strong.  IMHO, this is the difference between being a programmer or developer, and being an engineer (or scientist).

Having a good handle on some of the more "exotic" data structures could be useful.  Trie's have some interesting properties (and a modified version called a hash array mapped trie is what powers clojure's map type).   Red Black trees are used in linux's fair scheduler, and are useful wherever it is desirable to have balanced trees.  I started implementing an RB tree in C++ to help me learn C++11, but I'm going to do this in either julia or clojure instead.  I might even attempt a B+ tree, though the little I have looked at them, they look quite complicated.

I could use some of the data structure knowledge to help with filesystems.  Lately, I've been getting interested in Big Data.  Not just the analytics, which is cool in itself, but in the implementation of Big Data.  For example, you have to store all that data somewhere.  One of the few places in the linux kernel with complicated data structures is in the file system.  Lately, there's been a lot of interest in software defined storage systems.  It isn't just computers that are getting virtualized....you have VLAN's and software storage too.  Distributed software based file systems like HDFS, Glusterfs and Ceph have been gaining a lot of interest.  It would be interesting to see what it would take to build a file system.

While filesystems can enable Big Data, the other side of course is in analyzing that data.  One of the first things that got me into Computer Science was artificial intelligence.  It's one of the few fields that is math heavy (other than 3d programming or scientific computing) and that alone makes it interesting.  Everything in AI from support vector machines, decision trees, bayesian classification, markov models, neural networks and genetic programming is fascinating.  Given that julia was tailor made for scientific computing, I think it would be very interesting to convert the code samples in Machine Learning in Action book from python to julia.

Somewhat related to data structures and filesystems are graph databases.  A graph can simulate any other data type (though not with the same runtime or space complexity), from arrays to hashes to trees.  Ironically, they are better at modeling many kinds of relationships than traditional SQL databases, and much better than document DB's like couchdb or mongodb.  Graph databases would require a filesystem (for persistence), a hypergraph data structure, algorithms to walk the nodes (graph theory), and a strategy for ACID.

Messaging frameworks are a necessary component of any medium sized or above complexity project.  They are replacing more traditional RPC style communication styles due to their ability to be asynchronous and to do interesting things like pubsub or fanout communication.  With the proliferation of virtual machines and even virtual storage, data has to be shuttled across networks more than ever.  At first glance this doesn't seem very Comp. Sci related, but the key is in the implementation.   Using coroutines and asynchronous IO is the key to this problem.

The parser combinator may seem like an odd choice, but think about what is required to build or design one.  It requires understanding lexing, scanning, and therefore building DFA's.  There's also the aspect of how to make this functional in design, so that you can chain them together to use it as a recursive descent parser.  Writing a parser combinator will to a large degree be a stepping stone to writing my own language.

So why will I try to implement almost everything in either julia or clojure?  Clojure is starting to gain momentum with other companies and while I think most Java developers would rather graduate to Scala since it's a little more familiar, I think clojure will find its own niche.  For example, Puppet Labs recently ported some of their technology to clojure from ruby.  There's also a project at Red Hat called Immutant which allows you to write cloud apps in clojure.  And jclouds supports not only java as a first class citizen, but clojure too for your open cloud needs.  Julia is a very intriguing language with a lot of potential that I will shortly write a blog on.  The devs actually recommend learning julia by looking at the source code for julia itself.  This may also give me some insight into how LLVM works, since julia uses LLVM as a jit compiler.

How functional programming has motivated me to relearn math

Ok, I will admit it.  The first thing that attracted me to learning a lisp dialect was the coolness factor.  There's a saying about being a Smug Lisp Weenie.  To wit, we have this chart that I first saw several years ago:
http://lukewelling.com/wp-content/uploads/2006/08/programmer%20hierarchy.pdf

Actually, this chart was made by someone who only works with "pure" software programmers.  Whomever made this chart must have never run into the HDL (eg Verilog or VHDL) guys, as they would most prominently have been at the top.  Nevertheless the point is well made that other programmers feel that Lisp programmers feel they are superior to everybody.  Just read some of the reviews on Amazon.come for the classic SICP book by Abelson and Sussman and see how some people feel about lisp (or scheme) programmers.

I felt like there must be something about Lisp to inspire such a love/hate dichotomy.  For those who hated it, was it just a case of sour grapes?  Those who couldn't "get" lisp simply derided it as a defense mechanism?  My first experience with lisp was...well, confusing.  We had to do a little bit of lisp programming at my university's Artificial Intelligence class.  To say I had no clue how lisp worked was an understatement.  I mean 'car' and 'cdr'?

Regardless, a few years later (ca 2010), there was this new language Clojure that everyone was talking about.  I also recall reading around that time this hilarious article http://www.secretgeek.net/lisp_truth.asp.  I remember busting out laughing over and over, even though I hadn't yet grokked any lisp dialect.  Surely, a language that could inspire a page as funny as that had to have something go for it.  Not to mention that lisp is the 2nd oldest continuing language around.  So I decided to dip my toes into the Clojure waters and see what all the fuss was about.

It was hard.  Real hard.  I mean, how can you have a language without assignment?  Without for loops?  Where there isn't even technically a variable (as the oxymoron "constant variable" would surely indicate)?  It quite literally took me months to wrap my head around this bizarre concept.  To be honest, I think a good deal of the difficulty of the language wasn't so much around the "lispiness", it was was around the functional aspects.  Lisp and Scheme aren't pure functional languages (technically, neither is Clojure, but it is moreso than either Lisp or Scheme as all data is immutable and most sequences are lazy).  The idea that lisp is stupid because of all the parenthesis is....well stupid.  In fact, it's what makes lisps so beautiful and magical.  Lisps are already almost in their raw abstract syntax tree.  There's also very little syntax required to learn.

But what does all this have to do with math?  Functional programs are very closely related to the idea of mathematical functions.  The Fortran and Algol descended languages have at their core the idea of a Von Neumann machine.  The language they model closely resembles the hardware it runs on.  This imperative approach of modifying registers or memory is basically what Turing machines are about.  If i ever make gobs of money, I will try to buy an old lisp machine.

But I digress.  Lisp was really designed as a symbolic evaluator, and rather than follow the Turing/Von Neumann model of computation, it took the lambda calculus as derived from Alonzo Church as its basis for computation.  Although turing machines and lambda calculus are functionally equivalent (they have the same computational power), their expressiveness is worlds apart. 

I recently tried looking at some papers and books on denotational semantics and the lambda calculus, and I realized just how much math I have forgotten.  So, as bathroom reading material (yeah yeah...too much info), I have my old discrete math and formal languages and automata theory books that I can peruse.  I'm amazed how much I have forgotten.  Things like modular arithmetic, bijections, counting (set theory), equivalence relations and so on.

This is the heart of computation.  While most software engineers are content with learning the new fad of the year (hmmm, hadoop seems popular now, or maybe I better learn objective-c so I can write a bright and shiny iPhone app), I am more fascinated by the foundation of computation itself.  But it's also hard.  In all honesty, learning some new library is really not that hard.  What interests me more than programming with hadoop...is the science behind hadoop itself.

And it all starts with math, and lots of it.  It's not easy.  I remember Discrete Math to be insanely hard.  I had to retake linear algebra and abstract algebra.  Automata Theory wasn't a walk in the park either.  And learning all this math helps a software engineer design better algorithms or come up with more novel ways to solve a problem.

Now that the holidays are over, I'm going to resume my data structures project, and maybe tinker around with some other CS fundamentals.

Veterans and Memorial Day...a strange feeling

This holiday always brings out a strange feeling in me.  While Memorial Day is about remembering the fallen and is a somber day to reflect on sacrifices human beings have made for their fellow man, Veteran's Day elicits a mixed feeling in me.

My grandfather was a Captain in the Navy, and 3 of my great uncles all served.  Even my great aunt Kaye served as a secretary in the Pentagon and with the government before it was built.  My other grandfather was not in the military, but he and my oldest uncles sometimes used to go Japanese head-hunting at night as part of the Filipino resistance.  When I was a younger boy, I always wanted to know about my grandfather's war stories, but for some reason, I never asked him even though I was very proud of him.  To this day, I measure myself not to my father, but to my grand father.

But neither did my grandfather or great uncles tell me about WWII.  Perhaps they thought I was too young, or maybe they just didn't want to tell the stories.  My father never knew (or never told me) about their experiences either.  And that was the great mystery.

As a young boy and teen, I had my adolescent fantasies of war, but perhaps not like others.  Where others might have seen glory in war I did not (as Robert E. Lee once famously said, "It is a good thing war is so terrible, otherwise we should grow too fond of them").  As a teen, I was fascinated with the Vietnam War.  The more I learned about it, the more ashamed I became of America.  How could we deny a people the right to a democratic process?  Did we honestly believe we were going to stop Communism and its "Domino Effect"?  How could we treat the returning soldiers so cruelly and coldly?

I remember as a teen reading the book "Once a Warrior King", and the author had both of his legs and one of his arms below his elbow blown off.  When he got back home from his devastating wounds, a man asked if he had served in 'Nam.  The author said yes, and the man said, "serves you right".  The amount of hatred and rage that boiled inside me upon hearing this was surprising, even to me.  I wanted to literally kick the man in the gonads until he could no longer produce offspring.

About this same time (mid to late 80s), I was into playing role playing games.  I had already graduated from Dungeons and Dragons at this point, and was playing a Post-Apocalyptic game called Twilight 2000.  I was also running a campaign using the Phoenix Command rules in Vietnam.  By this time, I had already read several books on the Vietnam War.  Even one of my "hippie" aunts and uncles gave me a "Bright Shining Lie"  by Neil Sheehan to read.  The movie Platoon had also come out around that time and fascinated me.

But my roleplaying sessions were not like a youthful idolatry of desiring to be a soldier.  Despite my age, I tried to inject the senselessness of the war, of comrades dying, and making the other players having to face the moral dilemma of shooting children coming at them with grenades.  I remember watching Platoon with the scene where Sgt. Barnes has a gun to a little girl's head and a tear coming to my eye.

So if war seemed to terrible to me, why in the world would I want to re-enact it?  Why was I seriously considering going into the military (early 90s)?

Desire.

Yes, desire or at least infatuation.  I wanted to know the wisdom that these men like my grandfather and great uncles had learned.  As Aeschylus said in Agamemnon (taken from http://wist.info/aeschylus/6209/):

Wisdom comes through suffering.
Trouble, with its memories of pain,
Drips in our hearts as we try to sleep,
So men against their will
Learn to practice moderation.
Favours come to us from gods.

These men saw things and experienced things that I will never truly comprehend.  I do not wish to see death or have to kill others.  Not by a long shot.  But the idea that these men had a secret knowledge that I did not used to consume me.  How could I be truly wise unless I experienced suffering like they did?

But as I got older, I realized that I can experience the pain and suffering of others without having to go through an awful war.  Perhaps that is why I was attracted to Buddhism.  All other religions attract you because of promises of eternal life and salvation.  Not Buddhism.  Buddhism offered only one thing: to be free from suffering.

It is interesting that the only path to Buddhism is through suffering.  And once you have crossed the stream just a little bit, you will see that others pain is your pain too.  It is immense and hard to bear at times.  But then I think about what these veterans have seen, and what little suffering I have experienced pales like a stubbed toe in comparison.

So here is my salute to all veterans, past and present.  It is a strange thing to be in a profession where you hope your skills are never used.

What to study...real world pragmatism vs. haskell, scheme and CS fundamentals

So there's a lot of stuff to catch up on mainly because I've been very busy.  I switched jobs back in August and moved to Salt Lake City, Utah.  I went back to being an Automation/tools engineer rather than doing linux driver development.  Why?  I'll go over that in another post as it deserves its own topic.

Currently, I'm mostly doing some heavy duty and advanced python (metaclasses, lots of decorators, import machinery hooks, and an asynchronous socket based message system), and pretty soon I'll be working on some C++ code as well.  I've been working on some messaging frameworks too (MOM).  And lastly, I'm still plugging away at some computer science fundamentals.  I haven't touched my data structure project in awhile due to the business of moving and switching jobs, but I intend to get back on it.

In fact, I've been reinspired of late by reading a couple of books.  The first is the classic SICP.  Although I downloaded a pdf version, it's only the last few days I actually started reading it.  I finally got to the part about Normal Order evaluation versus Applicative Order evaluation and I had never considered that before.  In addition to the data structures project I have in C++, I figured I'd start working through the SICP problems and solve them either in Racket or Clojure.

While learning more about Scheme, I came across an interesting tutorial/book called  "Write Yourself a Scheme in 48hours" which is a guide on how to make a scheme in haskell.  I started reading that after finding Real World Haskell and Learn You a Haskell online.  I just started getting into Haskell a few days ago, so the syntax is a little odd to me still.  Nevertheless, I found it less confusing than 2 or 3 years ago while looking at it.  Haskell seemed to come up a lot while I was reading different forums or blogs on Clojure as there seemed to be quite a few comparisons.  I still don't quite understand Monads, but what I have seen so far is pretty interesting.

The idea of writing a Scheme in Haskell intrigued me more and more.  It seems like there is no language with the set of features that I'd like:

• immutable data structures by default (not in Racket)
• "native" continuations (not in haskell or clojure)
• lazy evaluation by default (not in clojure, possible in Racket)
• STM implementation (AFAIK, not in Racket)
• lisp style macros and homoiconicity (not truly available in haskell)
• C(++) ABI compatibility (none with c++, hardest to do in clojure )

The last bullet point is in all honesty, probably the hardest, but should be something allowable by using LLVM to build the language.  Building a Scheme in LLVM would be no small undertaking.  C++ is a hard enough language by itself, and then learning compiler theory and the LLVM libraries on top of that would be quite a challenge.  I could in theory use Boost.Spirit (which is a C++ parser combinator library) to build the lexer/parser part for the language, but there's a part of me that wants to learn how to make my own parsers.  But if I'm willing to forego that last bullet point, I could make a Scheme variant written in Haskell, and it would have the best of both worlds (pure functional procedures and data structures, lazy evaluation, STM, and homoiconic lisp style macros).

While all of this stuff is fascinating, in the back of mind, there's a nagging little voice that keeps saying, "but is this going to help your career?".  Even though all these topics are fascinating, I do sometimes wonder if I wouldn't be better off learning Ruby, Hadoop, some web skills or brushing off my Java.  Afterall, that's what employers want.  As I've heard many managers say, they want a candidate that can "hit the ground running".  This is a sad and unfortunate failure of a corporate culture to think this way.

Literally the last day that I was working at LSI (my old company), I got an email from a recruiter at Google saying he would like to do an interview with me.  I was tremendously flattered (not to mention mad at the universe for the quirk of fate that the opportunity didn't present itself a month earlier).  About a week later, I got a similar offer for a job interview at Amazon and Facebook both.  Although these opportunities were amazing, even had I still been searching for a job, I knew I was not ready for none of these technical giants.

I am not a bad software engineer, but I also believe I lack some strength in fundamentals.  What do I mean by that?  Software engineering is about more than knowing the intricacies of some programming language, or being a guru at some popular framework or library.  IMHO, there's a vast gulf between being a programmer, and being an engineer.  Programmers can churn out code.  It can even be well thought out and planned code.  But it takes a different mindset to know _how_ to tackle a problem in the most efficient and maintainable manner possible.

For example, what are the run time or space time constraints for some algorithm?  Given the choice of using some data structure, how will multiple concurrent threads access that data structure?  How do you deal with constantly changing inputs to a program, do you write a regex, a parser, or just rewrite your wrappers everytime?

There's a reason Computer Science majors study the basics of computation.  In fact, while studying data structures and algorithm analysis is interesting and useful in the real world, there's still far more to it than that.  For example, Graph Theory is heavily used to solve many problems, from version control systems to finding paths in a complex data structure.  Perhaps it may seem a waste of time to learn about finite automatas, but without them, how do you build regular expressions?

Some might argue that the hard work has already been done.  Just grab a PCRE library or some built in library or data structure in your language and do some real work.  I've even heard some managers complain of a "research project" mentality.  The essence being, if you're not coding, designing, scoping or documenting, you're not working (ie, there's no room for research).  I've had quite a few engineers tell me I have a much different philosophy than they do about solving problems.  My philosophy is that if I already know a way to solve some problem that would take one hour to implement, but I heard that there's another possible more elegant or more efficient way to solve the problem, but it would take one day to learn and implement, I'd much rather take the day to implement it in the more efficient or elegant way.

Now, the trick is, where's that cut off in terms of practicality/pragmatism and engineering elegance?  Management would usually say it's better to implement the rough way first, and then improve it later.  Afterall, that's the Agile mantra isn't it?  Get something working first, and then make small modifications over time to improve it.  The problem is, that is a false dichotomy.  The truth is, you rarely have time to ever go back and improve upon a solution.  So the old crufty but usable implementation is what stays forever more.  As my mom and grandfather used to drill in me, "if you are going to do something, do it right the first time".  But that doesn't mean you have to build some grand cathedral or not at all.  You can still spend the extra time to learn the better algorithm but implement it in a simple style.

As a simple example, you might be tasked with searching for a value in some data structure.  The simplest solution is to create an array and then just search element by element.  Simple right?  But why not store it in a binary search tree instead?  Just a tiny bit more difficulty, but now instead of a O(n) search time, now you have O(log n) search time.  A self-taught programmer might not have any clue about binary search trees (I know I didn't when I was teaching myself C++).  For a more real world example, if you have a relatively complicated piece of a formally structured data, do you solve the problem with a regex or with some other parser?  Take for example the typical example of the get long options in C (and its variants in other languages).  In python for example, you might have "-f 10", "--foo 10" or "--foo=10" (and it might even have a default value so it may be missing entirely).  So how do you come up with a way to verify all the command inputs are valid, or to programmatically come up with all of them?

There is perhaps a darker side to that nagging little voice in the back of my head that tells me I should be studying some popular web framework.  On the one hand, I would eventually like to move back closer to my family in Florida.  The problem is, Florida's job market is pretty much either as a web guy (back or front end), a database guy, or something in the military industrial complex.  While none of those are particular bad, I don't have the skills in web or database technologies, and the military industrial complex is very fickle from what I have heard.  The other insidious little aspect of this annoying voice is that learning frameworks and libraries is the easy part.  From my experience, it just takes time and some practice to learn some framework.  But learning the fundamentals of CS is _hard_.  In fact, I've never been as challenged in my work career as I was in school.  I've often found that an interesting conundrum.  Why isn't work as hard as school?

So despite that little voice, I will continue on training my fundamentals.  I remember many years ago, I was taking a class in Choy Lay Fut, a "modern" kung fu style that was amalgamated from several other styles during the mid 1800s.  For two months, literally, all I did was stances; horse stance, bow stance, cat stance, etc.  But I knew why the sifu was doing that.  Without a solid foundation, it is easy for your opponent to take advantage of you.  Everyone wants to get to the kicks and punches and blocks, but without a solid grounding, it's all useless.  As another martial arts analogy, I remember while taking Aikido, we were all told to breathe in, relax and in a slightly low posture, the sensei would come by and gently push on the center of our sternum.  It was easy to resist.  He did this several times, and then one time, he very lightly tapped us on the forehead, and a feather weight touch to our sternum toppled us very easily.  Lesson learned....your foundation is all important, and don't let anything distract your mind.

Something like OSGi for python

Ok, so I have been knee deep into python again for my new job as an automation engineer at Fusion-io.  Although python is a nice language for small to medium sized project, when you start getting into the middle tens of thousands of lines of code, python's "convention over bondage" philosophy starts to wear a little thin.

One problem I have been thinking of tackling is creating an OSGi like component based framework for python.  If you aren't familiar with OSGi for Java, it's a modularity system for Java that among other things, helps resolve CLASSPATH issues, versioning problems between jars, dynamic lifecycle of modules, and helps design more modular and reusable code.  Perhaps you are thinking that python already has modules and packages, and this isn't necessary.  But hold on, let's examine this little problem in more detail.

It's quite common in the Java world to hear the expression, "program to an interface, not an implementation". Java, lacking multiple-inheritance, has Interfaces to help resolve that problem.  But an interface's true strength is that it enforces the set of behavior that something should expose.  The idea is that the implementation should be a black box to the client programmer...he just wants the functionality exposed from the service.

You might for example have many different ways of retrieving an object over the net.  You might use FTP, or SFTP, SCP, http, or one of many ways.  Even amongst these choices there are subchoices.  Do you use python's built in ftp library?  Or do you have some ftp client utility that you want to wrap using subprocess?  Do you see the problem now?

The implementations change, but they all have a set of common functionality; the interface.  That is what you program to...the interface...not the module or class that exposes the concrete implementation of some functionality.  Being able to code against an interface allows you to swap out the underlying implementation.  Perhaps you are wondering why you would need to do that?

It is a sad fact that requirements are always changing or software breaks.  Perhaps some tool you are using changed the output and you were screen scraping, or the function defintion to some remote API you were using changes.  By decoupling the interface and the implementation, it eases these kinds of problems.  It also allows you to create an extension or plugin based system.

I can already hear some people say that python has no need for a plugin system.  Its dynamic nature makes plugins a "built in" feature of the language.  That's true, but it's also like the wild west.  Even python itself realized this with the creation of abstract base classes.  By creating an ABC, it allows a python developer a way to specify that a subclass must override and implement a set of methods.  If you use monkey patching to "plugin" new functionality, there's no way to guarantee that whatever got monkey patched is honoring the interface's contract.

Let's consider a more concrete problem.  Look at the example dependency diagram below:

So we have a module Work.py, that has a dependency on Command.py and Logger.py.  The problem is that Command.py also has a dependency on Logger.py, but of a different version.  At some point, Logger went through some kind of API breaking change, and Work uses the older version, but Command uses the newer one.  How can you load the Logger module in python?

PYTHONPATH won't work (nor with a .pth file).  Neither will locally installing Logger.py to different locations (instead of site-packages).  You can't for example, sys.path.extend(["/path/to/1.1/Logger.py", "/path/to/2.0/Logger.py"]).  Neither pip nor setuptools will help you because they will simply overwrite the module in site-packages (at least, AFAIK).  And finally, virtualenv or python3's new venv won't help either.

Why will none of these "standard" solutions work?  The problem is how __import__ works and how it loads module objects into sys.modules.  I'll be speaking here from a python3 perspective, as this is what I have done my research on.  It will be highly beneficial for readers to look at the python docs for the import system and importlib.  When a program calls import, three  basic operations are done:

1. Find the module
2. Create the module object
3. Bind the object into a global module namespace

 The first two are handled by the builtin __import__(), and the last is done by the import statement.  So the first thing python will do is look in the global namespace to see if the symbol is defined in the global module namespace.  If it finds it, it will return it.  This global module namespace can be seen by printing out sys.modules.  The key to the problem is in #3...python only has a global namespace for modules.  If you have two modules with the name Logger, the second time you try to load Logger, you will in fact get the first one.  Aliasing the module with:

    import Logger as Logger_ver2

Will not work.  So is the problem solvable? Fortunately it is.  You can use either the importlib or imp module to get around the problem, though it is a bit odd.  The key to understanding how this is works is in knowing that the import system uses Finders and Loaders.  Finders do the job of determining where to look for and check if a file really is a module (or package).  If the find is successful, this class returns a Loader object, and this Loader object actually returns the module object so that it can be used.  Using this method, you can give a name for the module as something other than the filename.  So you could for example load the module into sys.modules["Logger2"].

import importlib.find_loader

loader = importlib.find_loader("Logger", path=["/path/to/1.1"])
loader.name = "Logger1"  ## necessary since sys.module is a flat namespace
logger_v1 = loader.load_module()
logger = logger_v1.get_logger(__name__, "DEBUG")

loader = importlib.find_loader("Logger", path=["/path/to/2.0"])
loader.name = "Logger2"  ## change module name here too
logger_v2 = loader.load_module()
logger2 = logger_v2.get_logger(__name__, "DEBUG", "/path/to/logfiles")

The solution is a bit ugly, but it is possible. A more elegant solution would be to derive your own PathFinder and Loader classes.

However, this is just one small part of what a component based framework would have to provide. For example, how do you encourage programming to an interface and not an implementation? Isn't duck-typing enough? Sadly, no, it's not. Remember, an interface doesn't just specify the function signature, it also encapsulates the "what". An interface is supposed to hide the "how", but just because two classes both have a run() method, one class might mean that as launching a new process/thread and for another class it might mean something as totally different as telling the same process to start. In other words, a good interface doesn't just mimic the same function name, arguments and return type. It also exhibits the same behavior. How that behavior is achieved is irrelevant, but we need to make sure that everyone is saying the same thing.

Then there's the matter of dependency injection. Fortunately, this is much easier to do in a dynamically typed language like python, but it's still requires some thought, and what exactly a person means by "dependency injection". While decorators are powerful, they lack one crucial feature. Decorators can modify what happens before or after a function call. It can modify arguments to the function call. It can even replace the wrapped function entirely, or not run it at all. It can change the return type of a function. But what it can not do is modify what happens _inside_ the function. Unless you have the luxury is developing in a lisp dialect with their uber powerful macros, this will remain out of reach. The closest you can come to achieving this is to alter the AST of the function which requires some pretty deep knowledge of python internals. So how can you "truly" insert the dependency into a function? Fortunately, one thing decorators can do is change the arguments and even environment of a wrapped function.

Another aspect of many component based frameworks is the ability to dynamically add or remove components during the lifetime of a program. Any plugin system worthy of the name must be able to insert a plugin and provide features the plugin exposes. A more difficult matter is how to remove a plugin as other plugins may now have a dependency on another plugin. So a graceful removal of all plugins must be considered. Here again though the module system of python leaves a few things to be desired. The sys.modules acts as a cache and while it is possible to reload a module, entirely removing a module from the namespace is more tricky than just deleting the entry (afterall, sys.module is just a list).

Frankly I'm a little surprised at these "enterprise" features in python. While it's possible to do all the above, it requires some work. Indeed, there have been attempts by other python projects to provide the basis for these kinds of frameworks. Perhaps over time, I can contribute myself.

So, how do you say "kowtow" in Mandarin?

So, on Thursday my order for Fluenz Manadarin 1+2+3 came in and I started it yesterday.  So far, it's pretty interesting, but I'm only on lesson 3 so far.  This is the first long (2hr) lesson and the first two were relatively short.  It's a bit expensive at $358, but I purchased it thanks to a rather large reward I got at work.  I had to do a bit of deciding between it and Rosetta Stone, but the reviews were hands down in Fluenz's favor.  

One rather big difference between Fluenz approach and Rosetta Stone's approach is that Rosetta Stone teaches by the immersion method, which seems to be good for children.  However, Fluenz believes (from indications in neurological science) that adults lose the "plasticity" of the brain that makes children such rapid language learners.  Unfortunately, when I was growing up, the prevailing attitude was that it was harmful to teach a child multiple languages as it may confuse them.  This is entirely backwards, and the best time to learn a language is as a child.  So Fluenz took a different approach which for lack of a better term is the "explanation" approach.  Rather than lots of memorization or showing lots of pictures (immersion method), basically things are explained to you as you go along.  So far, I've found their approach fairly easy. 

The biggest criticism of the Fluenz program is it's lack of teaching of Chinese ideograms.  I remember when I took Japanese in college, by far the hardest part was learning the Kanji (the Japanese version of Chinese ideograms).  Learning the stroke order and radicals was no fun and could literally take up 2+ hrs a day in homework.  In Fluenz, everything is in Pin Yin, which is a romanization and phoneticization of the characters.  In other words, it is teaching you to speak and understand, but not really read or write.  But I think that's ok for now.

I probably should have learned Japanese or even Tagalog, but I think Mandarin is going to be a very important language in the next few decades.  And who knows, maybe I can find a job in Hong Kong or Singapore.  You never know, maybe I could even work in Shanghai or somewhere in Taiwan (yeah yeah, I know, Taiwanese mostly speak Hokkien, and most people in HK speak Cantonese).  Regardless, knowing Chinese will open up some opportunities in my career.

It may even open up some possibilities in my understanding of Buddhism.  Many Buddhist sutras were only saved in Chinese when Buddhism was essentially chased out of India.  It's unfortunate that many people don't understand how much of the meanings of their religious scripture(s) has changed due to elements being lost in translation.  This is especially bad for the Bible, which went from Aramaic to Hebrew to Greek to Latin to English.  You almost have to wonder how much of the translation is left.  For Buddhism, I'd like to re-read some of the Sutras in the Chinese form and see what it is like.  Of course, that will require reading comprehension which this program doesn't teach.  Still though, it'd be cool if I could read Buddhist sutras in Chinese (and from what I understand, even though Cantonese and Mandarin are pronounced differently, they can each read and write the same).

I consider myself pretty good at learning new computer languages, so let's see if this applies to human languages as well.  The one thing that worries me is that I don't consider myself all that good at memorization.  In fact, I have a pretty horrible (short term) memory.  I've always said I was extremely grateful that almost all the classes in my Computer Science curricula were open note.  I think I am pretty decent at learning new concepts, but my memory is not so hot.  And of course, learning new vocabulary is essentially 100% memorization.  The thing with human languages is that you can't think like you do in a programming language.  In a human language, you don't translate from one language to another.  I always used to say in French, "Je dois traduire les mots francais dans les mots anglais. Je ne pense pas vraiment en francais".  Basically, you shouldn't translate the words in your head from english to the other language and vice versa.  It has to come naturally and without an internal translation.  That requires more than memorization, it requires making it a part of yourself...and for me, that's difficult.  

The other thing I am a little fearful of is pronunciation.  Mandarin is a tonal language with 4 tones.  Basically this means that the meaning of a word changes by how it is pronounced.  I have been complimented before on my french pronunciation, so I am hoping I have a good ear for understanding.  However, speaking the proper tones back might be a little harder.