Blogging for ACM !

Thanks to the recommendation from Max, I've been given the opportunity to be an official blogger for CACM. My blogs appear at BLOG@CACM and the first post on the OOPSLA cloud workshops ,Taming the Clouds at OOPSLA, is available now.

IBM, Linux and Research

My internship at IBM Research came to an end about two weeks ago (on July 31st). It has been an exceptionally good one and I've enjoyed every minute of it despite the fast moving schedule. I have been given great opportunities [Like being featured in the home page of W3, IBM's intrarnet :) ] and I am really happy about the outcome of this internship in terms of the result and the future collaborations we managed to forge.

From Random

This is the memory of the last day, just before leaving where I had to submit my badge and the laptop.

In summary, following are my observations on IBM, the environment and the culture. IBM here onwards refers to IBM research, particularly the Almaden Research Center when not explicitly stated.

Linux @ IBM

My computer in IBM ran Linux (Ubuntu 8.10). Last time I had to do a similar thing, (in 2007) I struggled for 2 days to get Lotus and other IBM productivity software installed and configured. This time to my amazement, all I had to do was to download a single shell script and run it! it updated my repositories, installed all required IBM software, provided hints to configure them and installed shortcuts to help and support groups for the flavor of Linux I am running. Linux has become one of the mainstream operating systems for IBM and there is ample support inside IBM if you decide to live the Linux way of life.

There were some pain points though. Wireless drivers for T60 thinkpad I had [Atheros IIRC] sucked big time. IBM internal wireless uses LEAP and I was never able to configure LEAP or EAP to work properly [ WEP and open networks worked fine ] . I had a wired network so most of the issues came when I was in meetings.

The ATI graphics drivers did an ok job but that could have been improved. There is no support for ATI chips [ Well not from AMD ] for Ubuntu 9.04 and I did not want to upgrade primarily due to the fear of losing my nice dual screen setup.

Research @ IBM

In terms of research, IBM is an excellent place to be. Apart from the diversity of researchers, there is always this niche area that you can work on and there is the freedom to collaborate with other groups.

If you are used to academic style research IBM might present a difference for you though. While researchers are free to do exploratory research, that freedom is not 100%. At the end of the day research for IBM is all about improving their products and gaining/maintaining their market share.There is this expectation(implicit and sometimes explicit) that research would contribute to that cause. For example the cloud related research project I was part of, gained a lot of interest from all tiers of IBM since it seems to have the potential to put IBM ahead in the competitive cloud provider market. I cannot imagine that the same success would have been there if we were to do a different project that is of not so much business value, even if there was significant scientific value to it. On the other hand there is the need for constant marketing of your research idea among peer researchers and convincing executives that your idea is worthwhile which takes similar time and effort to writing grant proposals in academia. However I understand the need for this. At the end of the day money just does not drop from the sky :)

Culture @ IBM

One word description - "excellent" :) Being in the silicone valley, I suppose some of the habits of the young startups could have seeped through :) The freedom of working hours and location, the coffee hours in the evenings [ No free food though. Even the evening free coffee is meant to produce a social gathering ], constant list of guest speakers and experts delivering excellent lectures , great research library , there is nothing you can complain about :) [ to tell the truth here, some IBM veterans complain about the food. There is no restaurant you can reach without driving for 10 minutes and you end up eating at the cafeteria for the most part. The cafeteria people try hard to introduce variety into their menu, I've seen Italian, French, Mexican and even Indian entres. I personally did not have any issues with food, except for the fact that sometimes its too much to eat :)]

Last but not least, IBM Almaden is located in the middle of the mountains, a place with breathtaking scenary (see the map and some of the pictures I've taken) and somewhat ironically reminds you everyday that there is this vast space for you to explore as a researcher.

View Larger Map

From Random

From Random

Getting America to embrace the new age

I'm happy to see this news, the car exchange program to encourage people to buy newer (and more fuel efficient) cars. It is hard to imagine that US oil consumption would go down without such programs. For example I regularly see very old cars on the road [no not the modified hot-rods that have a thirties shell and a 21st century drive train :) However the new configuration may be worse than the original in terms of fuel consumption ] and the truth is after a certain time period, a car becomes seriously inefficient if not maintained properly. So a newer, more fuel efficient car is always a better thing for the planet.

Another important thing to note is that these cars are going to be scrapped. Some countries that have age restrictions for vehicles (such as Japan) just push the cars out of the country and let them be sold in other countries. This is exactly what happens in Sri Lanka where many cars are imported from Japan. The laws have been revised to restrict older cars from coming through however and I hope the situation changes soon. Anyway, the point is that once you take a gas guzzler out of the road, this program makes sure that it'll not be used again since it does not matter where you emit the fumes.

Year of English and IT

I was impressed by the following video, when I stumbled upon it in facebook [ Thanks Chinthaka for posting]



Not having a money sucking war anymore lets Sri Lanka focus on economic development now and getting such a complement from Bill is extremely valuable. I have great respect to Bill (Gates) for what he has been able to do with the software industry [ Apple's Steve Jobs, famously complemented Bill on building the first software company when the world did not see software as the driving force]. As I've mentioned before, although I have an ideological disagreement with Microsoft model of business, it is not a reason to discredit Bill or Microsofts innovations in software. This video statement however includes a slightly worrisome piece although the overall message makes me really happy.

My concern rises from "Microsoft will fully support this effort". Following the footsteps of my mentor, Sanjiva Weerawarana, I also believe that the model that apparently worked for India does not work for Sri Lanka. Why ? India has a huge advantage of man power and we don't ! What would work for us as a smaller but highly literate (thanks to the free education system that I am also part of) country is the Irish model. Irish are known to provide high level, specilaized software services (see their software portfolio) and known to have a highly skillful workforce. When I was still an undergrad, Sanjiva did a presentation on this and what immediately grabbed me is this idea of having high quality specialized services. I suppose WSO2, my old employer, achieved some of that success for SOA since right now if you are thinking SOA, WSO2 stands out as a major player [ Guess who Microsoft selected to demonstrate their SOA interoperability ? Not IBM, Oracle or SUN but WSO2 ]. WSO2 primarily consists of Sri Lankan talent. I still affectionately remember when we used to be a one floor office and you could point to experts in any of the WS-* spec areas , like "that's the WS-RM expert, that guy on the corner is the WS security expert" etc :)

Anyway - what is my concern ? I believe Free and Opensource Software (FOSS) has a great role to play in pushing Sri Lanka as the IT capitol. Having a Sinahala enabled Vista version is great, I would even consider getting that for my mom and dad who struggle with the english menus. But an influx of native language enabled Microsoft software that needs to be paid is not necassarily the best path to success. I believe that native language enabled Linux and other FOSS based software development methods would be the key in this effort. Why ? It comes free of cost, no strings attached and doing so will create a whole new specialization for us. Microsoft being a major support in the IT effort (and the introduction of the native language enabled Windows version) makes me think that the government could lean towards focusing more on Microsoft specializations. FOSS not having a strong financial backing [We all know "money talks"] would make things harder also.

In the same light I remember a discussion I had with James Clark years back. He was telling us what happened to one of the Linux popularization programs his Thai Opensource foundation did. They distributed linux preinstalled laptops at subsidized prices and people who received them just wiped out Linux and installed the Thai version of Windows (in many occasions pirated). According to James, the Thai version of Windows is pretty good and dominates the OS market in Thailand. I'm not saying something like that will happen in Sri Lanka but surely we will have to think of similar consequences if a serious imbalance is created.

We learn that relying on one cloud provider is bad - Once again !

yes - Many have predicted this and many others anticipated a similar scenario. However it happened when no one was expecting. I'm talking about the Google cloud outage that happened on 2nd July :)

There has been thorough investigations about what happened. There is a very detailed email in the user group [ read it here. Need to be part of the group though ]. Here are few things that got my attention. Here onwards sections in italic are extracts from the post mortem :)

8:07 AM --- The App Engine primary on-call engineer attempts to update
the System Status site with information describing elevated datastore
latency and error rates. However, the Status Site is only
intermittently available and returning errors on all updates.
Investigating the problem, the primary engineer discovered that the
isolated servers supporting the Status Site were running in the same
data center as the primary App Engine serving cluster. Thus, the site
ultimately depended upon the same GFS instance as App Engine itself.
The cause for this error in the Status Site was determined to be a
configuration error in the App Engine datacenter failover procedure.

This tells me no matter how deep we've been in handling large volumes of data, we are still not good at dealing with disasters! Such configuration errors come to light only in dire circumstances and its usually too late !

10:00 AM --- GFS SRE advises that the GFS engineering team has
identified the cause of the crashes as a "query-of-death" against the
GFS servers. Another user of GFS in the same primary datacenter as App
Engine is issuing a request to the GFS servers that reliably causes a
crash. The client was sending an improperly formed filehandle which
was not safely checked and sanitized by the server, and which caused a
stack overflow when processed. Now that it is known that the bug is a
malformed query from a client, GFS SRE identifies a MapReduce process
that is triggering the GFS bug, and the process is disabled. GFS
Master is no longer failing and GFS Chunkservers, which hold the
actual needed data, are starting to come back up by 10:30 AM.

Google is one of the most test oriented software companies. To understand how serious they are with tests you have to step into one of their rest rooms [ I personally have this experience twice ]. No matter whether you stand up or sit down to do your thing, there are well placed notices that tell you about the importance of testing and even little quizzes about "where is the bug". One would expect such test obsessed folks to generate near perfect code. An "unsanitized file handle" ? How could that have slipped through ? What this tells me is that we still have not perfected the art of creating software !

Most importantly this incident underscores what we have been talking about for some time. Dependence on one cloud provider. Google just proved that depending solely on them is bad !!

Twitter, Ruby on Rails and Jack Dorsey

Jack Dorsey [@jack] gave a talk at IBM research today. He came 'plain' with no slide deck but it was a great talk and Jack is an exciting speaker [ at least for me, he has the geek appeal :)].

IBM Research Talk with Twitter's Jack Dorsey from IBM Research on Vimeo.

Jack talked about the history, his obsession with couriers and dispatchers etc and gave an exciting answer to one of the questions I asked :)

Question: Do you think that your initial choice of technology effected the progress of twitter ? in other words If you were to do it again would you use rails or something else ?

Answer: [Not in the exact words but the intent and meaning is not modified. I did take a look at the recording. The time in the video are from 53.31 onwards ]

Yes. For example the use of Rails made the application accessible to people withing 2 weeks! The fact that people saw the application and reacted to it with such a short time was very important. Florian Weber , One of the Rails core committers was in the team (with close ties to DHH) and that also was enticing in choosing the framework.
The mistake we did was putting it all out there. We did not follow a controlled expose methodology and that is what came to bite us.
As for the technology perspective I would not have changed anything! We would be cautious and expect explosive growth but the platform would have been the same.

--

Obviously the actual answer is longer and the above is just the gist of it. However given that Twitter is the poster child for Ruby on Rails and Twitter reportedly had so many hiccups, it was surprising to hear Jack say this. His point was that because of the technology they were able to get the look and feel and the functionality rapidly and that made what is Twitter today.

The importance of this statement is that it is a realization of the wonder of rapid development with agile frameworks :) I have friends who believe that everything need to be done with basics without using packaged functionality. Also I've seen many efforts in the early phases of projects (sometimes simple ones that may not even go beyond a proof of concept) that focus on premature optimization and finally result in horrible code that is extremely difficult to comprehend. The sad part is that sometimes it is deemed that such complexity is necessary ['How are we going to efficiently use our memory then ?' ]. Jack answers all these 'fallacies' [ that is exactly how I see them ] by endorsing rails :) His point is that what is important , especially in socially driven applications such as Twitter is agility. Agility comes with more abstraction !

These days I work with Rails everyday and given the complexity of the application suite we (Max, Roy and me) are building, if we did not have Ruby on Rails, we wouldn't have done even half the stuff ! Its amazingly productive and given a good rails tool set (I use Eclipse Aptana studio - free version of course) it becomes a really good competitor to any Web development framework. And Jack just polished the chrome making rails shine more :)

How much is it gonna cost ?

While listening to a recorded interview with the defense secretary of Sri Lanka, I built the curiosity to find out what kind of funds are needed to sustain a war. We hear about terrorist organizations every day. What we don't hear or don't understand is the cost of sustaining their activities.

Amazingly it takes an enormous amount of money to sustain a war. From a quick search I gather that an RPG costs around $300 per unit and the launcher costs around $500. Given that terrorists can't just go to a store and order 100 RPG's they'll be paying much more than that in the black market. Let us double the price and count it as $600 for one RPG. Even a Westerner would just twitch on this number but you have to really consider the third world to see the enormity of these values. For example the gross national income per capita in Sri Lanka in 2007 is $1540. In other words one RPG costs a little less than half the yearly income of an average citizen!

These numbers come into perspective when you see the weapons caches the clearing operations turn out everyday in Sri Lanka. This report for example states that nearly 30 RPG weapons (which I presume launchers) have been recovered. From our calculations the launchers alone would have cost them $30,000 ! There has been numerous reoveries of weapons caches and they not only turn out RPG's - guns, ammo, satellite and communication equipment, explosives (claymore mines) etc are common findings. I'm sure if one is to count the complete cost of these weapons (I meant cost - not value - since when an illegitimate organization acquires such weapons I assume they have to go through the black market and pay more than the actual value) that were recovered, it'll easily dwarf millions of dollars.

Where can all this money come from ? As for the LTTE most of the funds came from extortion, drug trafficking and other illegal activities. I would assume for other organizations such as Al Qaida, it will be more or less the same means of fund raising. What most people don't realize is the enormity of these funds when they sum up. The Sri Lankan army once reported that the terrorists use a higher grade RPG (obviously more expensive) than the army version. The Sri Lankan forces never had the ZPU-4 type anti aircraft weapons that were recovered from the terrorist stronghold. So simply the terrorist challenged a government (of a country that ranks at 78th position in GDP, So Sri Lanka is poor but not dirt poor) in terms of their financial power, a power that came from summing up a number of illegal activities!