Monday, April 30, 2012

Making a Scheme...the grand plan

Man I hope I don't jinx myself, but there's a pretty good chance I might be working in a new position soon and I won't be an SDET anymore.  Lest Mr. Murphy come and visit me, I won't say what I might be doing until all the i's are dotted, and all the t's crossed.  But I will say that I will be doing a lot more low-level coding again.

So, it looks like once again, I'll be switching focus on what I'll be doing for my hobby time.  I'll be getting knee deep into linux internals again and need to brush up on my C.  I'm also going to spend a little more time looking at the Minix source code.  But especially, I'll be looking at LLVM and Scheme, specifically PLT Racket.  Why all of this?  And isn't Scheme a higher level language?

Let me start with why I am looking at Scheme now.  Although Clojure is a pretty cool language (especially the Software Transactional Memory, which I haven't seen an equivalent of in any other Lisps), it's still in Virtual Machine land.  And unfortunately, Java isn't all that great at interfacing with low-level C shared libraries or OS API's.  That's where Scheme comes in.  There are a couple of flavors of Scheme out there that have the ability translate into C code (for example Gambit Scheme and Chicken Scheme).  And although both of those Schemes look kind of cool, I am currently looking at PLT Racket (formerly known as PLT Scheme).

I'm even reading a bit of the R6RS standard, in order to help me wrap my head around Scheme a little better (Clojure, though a relative of lisp, seems to me to be neither truly a Common Lisp nor Scheme derivative...in essence, it seems to be its own branch on the lisp family tree along with language like Shen).  Although python is nice, and it is pretty easy, I want to learn a new higher level language that will let me interface with C more easily.  Scheme fits this bill, but it does have a drawback that I mentioned above....no easy concurrency support (and yeah, I have read that continuations can be used for a kind of parallelism, but I am not sure it can be used for concurrency).

So this is where LLVM fits in.  LLVM is a set of libraries which provides a front end (convert source code to the AST), optimizer and code generator (to generate the actual binary machine code for a specific architecture)  for a programming language.  LLVM is an interesting project, as it aims to help people write compilers, interpreters, or even JIT/VM's.  It can do so by providing a "universal" Interface Representation called the LLVM IR (I like to think of it as a universal assembly).  One could even create a language (it has lexers, scanners and parsers as well) with it.  And LLVM makes it trivial to call into the C ABI.

Do you see where I am heading with this?

R6RS Scheme standard
LLVM to create a JIT'ed language that can easily interface to C

There's one last piece of the puzzle...concurrency support.  Right now, the rage seems to be Erlang style message passing (Actors) for concurrency, but STM is gaining a lot of traction (Scala is implementing not one, but three STM libraries, haskell has 2 STM implementations, and pypy is experimenting with STM support).  I found a pretty interesting article by a C++ guru Bartosz Milewski on STM, including a link to an academic paper on Transactional Locking II.

Previously, I had toyed with the idea of implementing Clojure style syntax in D.  But the more I think about it, I realized it failed to satisfy several goals:

1) The stable dmd compiler only supports x86.  I want to work with  ARM processors
2) The front end to dmd2 is not open source
3) The LLVM based compiler ldc is not stable for D and is lagging behind
4) While it would make adding support for D modules easy, my real goal is support for C libraries
5) I would have to master D syntax and lisp/scheme

So I decided to something even harder:
1) Learn LLVM, including garbage collection and JIT byte generation
2) Learn how to make a lisp reader (it will be a LLVM based app )
3) Figure out how to implement STM in all of this
4) Slowly add in R6RS requirements (but not all)


Yeah yeah...I only have so much hobby time.  But creating a language is really something I've wanted to do for a LOOONG time.  I am getting close to paying off all my debts, which means I will be able to go and start on my Master's degree fairly soon.  And I really want to be able to design my own language.  Yup, I know, it's a dream of a lot of people, but I still think it would be cool.  But I also want it to be practical.

Some of the things that appealed to me about Clojure was that:
1) The syntax seemed easier for me to read that other lisps and schemes (more than just parens)
2) The appeal of built-in concurrent programming via STM
3) More scheme-like functional approach (immutability as the default for example)
4) Supports both a JIT bytecode generation, and a AOT compiled mode

So I definitely want to keep these in the toy language I will create.  But of course, there's one big hole in Clojure...good C/C++ library support.  Where Clojure can easily interface with Java, I want this language to easily interface with C (and ideally in both directions...but that might be too hard to support for now).  As a consequence, I want the language to support both a JIT'ed and AOT compiled (native) mode.

This is obviously a monumental task.  But I think it will force me to become a better Computer Scientist.  I will have to get better at many of the fundamentals of CS.  And yes, you DO need the things you learn about in school at work (choosing the right data structures, understanding complexity analysis to find poor algorithms, the ability to prove your solution is correct, etc etc).  Moreover, I am a firm believer that having learned several languages, and more importantly, different styles of programming has made me a better programmer.  When I was interviewing for my new position (at my own company), one of the interviewers noticed that I had Clojure down on my resume.  He seemed impressed and curious at the same time (he had heard of Clojure, and did a tiny bit of elisp, but thought lisp was too hard).

My feeling is that all the naysayers of lambdas in the upcoming Java 8 will eventually see their usefulness, instead of just decrying them as a "me too" feature for Java to catch up with C# on the bullet point list.  But, much to my surprise, many engineers are loathe to change.  I guess that's just one reason I consider myself a scientist rather than an engineer (engineers after all want stability, but scientists, in their quest for truth must be willing to give up the old in order to learn the new).

Sunday, April 1, 2012

A Testing Manifesto for Hardware companies Part 1

I was looking through some of my posts and was looking at some of the "drafts" that I never published.  I thought I had published this one earlier, but apparently not.  I wrote this draft about a year ago, but I thought it should see the light of day :)

...

I've been an SDET at my company now for about 3.5 years, and I've either seen other companies or divisions and their test strategies, or have talked to other SDETs and Test Engineers from other companies and received an idea of what their company's old test strategies were like.  I have since come to several conclusions regarding how testing is done, and how it should be done.  What I will write here is primarily of interest to the managers of Testing departments in hardware oriented companies as well as Test Architects, but engineers in the Test department should also find some use of what I shall say.

First off, let me begin with what testing should be:

  1. Even Hardware-centric organizations require enterprise techniques
  2. Hardware-centric organizations need to use fundamental tenets of good software engineering
  3. Use new but mature technologies suited for the task at hand
  4. Test Engineers are "true" engineers and should be treated as such
  5. Managers (Test and Development) need to understand what testing requires
  6. Don't mix up white box, black box, and acceptance testing
  7. Test Engineers and Developers have to work hand in hand
  8. Unit tests should be written by the developers (no "dev test")
  9. Requirements gathering should be an ongoing process
  10. Continuous Integration and Deployment is a must


Is your department not exhibiting some or all of these?  Perhaps you don't understand some of what I am talking about?  Or maybe (gasp), you think even if your department isn't exhibiting one or more of these traits, that it isn't important?  So, let me go into a little more detail into each of these issues, and explain why not following the above is harmful to your organization.  Then I will discuss just a few ideas on how to make sure your group is following the above.

Hardware oriented organizations don't understand enterprise level computing

Ok, I know, "enterprise" computing itself doesn't have a definition that's exactly entrenched in stone.  But if your SCM or Process group doesn't understand "Software as a service", "distributed computing", or web service technologies (or even doesn't understand remote services or remote procedure calls), then I submit that your organization doesn't understand enterprise level computing.  When  I say enterprise, I don't necessarily mean high-volume, high-transaction computing environments, but I do mean remote, distributed computing, with at least some level of persistence and data tracking/mining/relationships.

Even should you understand what enterprise computing is about, how would this benefit the test department?  Think about what a test group does.  It creates tests which are designed to expose defects in the hardware (or the software that controls the hardware).  There are many hidden assumption in this seemingly easy enough responsibility.  You should immediately think of the following  aspects to this:

  1. How are you reporting the results? (are you able to do data mining on results of tests?)
  2. How are users finding the test tools they need? (given a test case, is there a test tool for it?)
  3. How are users installing the test tool? (if they found the test tool, how do they install it?)
  4. How is a user supposed to know how to run the tool? ( Do you maintain elaborate documentation?  What arguments are supposed to be passed in for Test Case A versus Test Case B?)
  5. How are you discovering systems that can run tests? (Can you find systems not in use programmatically, so that you are executing 24/7?)
If your organization isn't linking TestCases to test tools, then I submit that you are in chaos.  If your organization isn't storing results of test runs automatically somewhere (hopefully a database of some sort), then you are missing great opportunities.   Being able to know what features in the hardware or software are associated with what test tool (and any other metadata required) is absolutely essential.  Think about what happens if you don't have this linkage.

Tester- "Hey Sean, is there a script for this test case I got assigned?"
Test Engineer- "What test case is that?"
Tester- "Ummm, let me see, it's ID 00716459"
Test Engineer- "Oh, that's the one to make sure the ioctl in the driver doesn't time out right?"
Tester- "yeah, but is there a program or tool for that?"
Test Engineer-"Yeah there is, let me go find the script on the common share drive"
Tester-"I already kind of looked there..."
Test Engineer-"Did you look under the Sean folder?"
Tester-"Yeah, but there were a couple of scripts that had similar names"
Test Engineer-"Oh yeah...you have to use the one with -version.1.3.5 in it"
Tester-"Oh ok."
Test Engineer-"And did you make sure you installed all the prerequisites on your test machine?"
Tester-"such as?"
Test Engineer-"Well, first you have to install..."


Clojure and OSGi/modularity?

Now that I am going full on with OSGi (and to a lesser degree, bone up on Java Servlets and maybe JavaServer Faces), I've been wondering about how Clojure will handle modularity.  Looking at the dev site for Clojure, there is a mention that Clojure should work with OSGi, and indeed, the Counter Clockwise plugin for Eclipse already must to some degree, but I noticed a few odd tidbits.

For example, if you look at the clojure dev site where it mentions modularity, it has an outdated reference to Project Jigsaw's scope (see here for a more recent coverage of Jigsaw's scope).  According to several sources, Project Jigsaw is not just an OpenJDK endeavor, but will be in the official Oracle release as well.  And some people are saying that the modularity aspect (combined with the long-awaited closures) will make Java 8 the most revolutionary release of Java to date.

Since Clojure's documentation is out of date, I wonder if the clojuremeister's are planning for how this new Java 8 modularity aspect might impact Clojure itself.  The good news is that if Clojure can support OSGi (and it kind of does thanks to clojure.osgi), then it should work with Java 8, since Jigsaw is supposed to be backwards compatible with OSGi.

What I would love to do is make several bundles in Clojure, and then let the framework take care of it.  I suppose I could just do an AOT compile of Clojure into Java, but the really killer app to me is if I could somehow embed a Clojure REPL as a bundle.  I might have to take a peak at CounterClockWise to see how they do it.  Hmmmm, a clojure NREPL, talking over websockets from a webpage....

All fancy ideas aside, all of this learning always comes at the expense of my "hobby time".  I suppose I should be thankful I have hobby time, but still, I'd prefer learning some other things.  I haven't done any clojure programming in awhile, and I don't want it to get rusty.  Heck, I never even got to practice a couple of things in Clojure (like multimethods or macros).  With regular POJOs, I can just take a java jar, and play with it in clojure.  But with OSGi bundles, I'm not quite sure how I can do that, since the bundle has to run inside the framework.

Anyway, something to think about.  I definitely want a "scripting" language for the stuff I'm doing (man I hate that word, and all the connotations it brings).  So if time permits, I'll see if I can eventually put up a blog post about writing an OSGi bundle in clojure.