Tuning replication in distributed datastores

What the heck have you been researching?  

Undoubtedly, this has to be the most common question that any research student is asked while at grad school. And I'm no exception. It has been a mystery to so many of my readers as to what I have been doing over the past few years. Unlike some of my other research projects, I now have something that I am allowed to showcase and let others use it too. Over the last year, I have been tackling problems that are of highest priority.  How fast can Facebook or Twitter be ? Can world of warcraft be more responsive and real-time ? ;)

Jokes aside, the crux of the problem is more pervasive than WoW or Facebook. What we are really asking is - how fast can interactive applications be? Can our web services be that extra second faster than what they are today? And the answer, I believe, lies in the fact that the application can only be as fast as the underlying data is fetched. This problem is exacerbated in geo-replicated databases which have users literally from all over the world and the applications that use these datastores are striving to be as responsive as possible. We (the shenanigans at our university) mulled over this for a while (a good year or so) and the solution was born - D-Tunes.

And the first version of D-Tunes (Datastore Tune), as a web interface is alive here. As a precis, D-Tunes configures the replication in your datastore in the most optimal way for your application, thereby ensuring that the application receives the best possible performance from the datastores.

For the brave, here lies the heart of D-Tunes in all its splendor. As the Spartans would have said, In details, lie the glory!  

The functional illiteracy called Geography

Years ago, when I moved out of India, I was amazed to find that most conversations in the western world always begin with the weather, something that we never encounter in the subcontinent. If I were to say this now in India - Looks like it is going to rain today, the most compassionate remark I can hope to get is - Are you crazy ? Of course it will rain - it is monsoon now ! And there's good reason to it too. Unlike in the west, weather is totally predictable in India. During summer it is hot, during winter it is pleasant (cold in northern parts), during monsoon it rains. There you go. As simple as it comes. 

The obvious reason for weather to be a hot conversational material (despite the numerous weather apps in the smartphones that people checked N times a day) is because it works as a wonderfully polite, safe, universally approved, social-awkwardness substitute. Elevators, Subway trains, birthday parties,weddings, B-DUBS, you name it. Any imaginable awkward conversation can be handled effortlessly by progressively suggesting weather alternatives - starting off with a mild excitement, transcending to a reticent amusement, or a disinterested observation, moving on to an affable annoyance before finally settling down on explicit bitching. It works - always. 

Extending this thought, I have been consciously trying to experiment with geography as a possible conversational substitute for weather. Over the years, here are a few of the priceless nuggets that came up during the conversation, those which I managed to remember or document. I have tried to present them verbatim to retain their natural conversational flair.

Wait! Korea isn't an island ? I'd always thought it was.

Yeah. Too bad.

I need to take a vacation in the Andamans. That'd be cool. The Arabian sea, I've heard is beautiful. 

You mean the Bay of Bengal, right ?

Same difference.

Really? Hmm. What is the capital of Assam?

Guwahati.

Nope. Its Dispur. 

Dude! You cant make stuff up. There's even an IIT there !

Japan to the United States is a long flight. 

Why ?

What why ? They are at the far end of the maps - that's why.

Which direction does the moon rise?

West, of course. Wait - that's opposite to the Sun right ?

So, if I need to get to Alaska, I have to catch a flight or take a ship right ?

Or you could drive. 

How? Isn't Alaska north-west of Canada ?

Yup. But Canada has roads too.

In the month of January, watching a game of cricket played between South Africa and India..

Why does the commentator keep talking about summer season games? Isn't that like another 5 months away. Brr...rrrr...

If I go across Antarctica, I should reach the Arctic circle right ? No ?

Do you know how many states India has now ?

Now, now. You're just getting political.

You know, the Musi river flows right beside Osmania campus..

He he ! Yeah., right.

The spread of South Indian culture northwards would have happened if not for the Vindhya ranges..

I dont think so. I remember reading that Agasthya had pushed those mountains back into the earth a long long time ago.

Malaysians are good badminton players.

Of course, they come from the Chinese sub-continent.

You mean, south-east asian

No, no. Chinese.

You know, India is divided roughly into two halves by the tropic of cancer.
The tropic of what ?

There are many more, but despite being free, online acreage is to be used judiciously. Jokes aside - the one thing that I want you to remember as a take-away is the value of questioning information that were fed into us as facts. Sun rising in the east is a fact - very few questioned, why? Fewer still, understood. True to the fact that humanity evolves and solves bigger and smarter problems, it remains to ponder what we traded off. A common example is the use of GPS navigators, an epitome of human innovation - replacing human directional skills. Good? Bad ? Your call. 

May our kids be able to spell Geography.

InvalidKeyPair.NotFound EC2 Boto package

It is fair to say that I have been acquainted with Amazon AWS for the past couple of years for my research requirements. Actually it is more than fair - it has been my bread-and-butter environment for scripting experiments on cloud platforms and wide-area storage networks.  And boto has been my go-to api for AWS tasks given that most of my experiments evolve from ad-hoc python scripts that does a bit of this and that.

And yesterday, I started receiving "InvalidKeyPair" response exception from the API out of the blue (not entirely true since I had made a host of changes to my scripts), which suggested that my keypair doesn't exist. A simple ls said otherwise - I found the keypair well and truly alive with the right set of permissions as well.

Source:

EC2Connection.run_instances(image_id='ami-09b11233', key_name='awskeys.pem', instance_type='m1.small',security_groups='default'])

Exception:

boto.exception.EC2ResponseError: EC2ResponseError: 400 Bad Request
InvalidKeyPair.NotFound. The key pair 'awskeys.pem' does not exist

After almost running out of hair to pull and Googl'ing all abstract keywords, here is the solution:

EC2Connection.run_instances(image_id='ami-09b11233', key_name='awskeys', instance_type='m1.small',security_groups='default'])


Apparently, the key is just "awskeys" and not "awskeys.pem". I haven't yet figured out why this exception wasn't raised until now or performed a trace-back to see when this started becoming an issue, but wanted to make a note right away just in case this is useful to someone else too.

Issued in public interest to all AWS dependent developers. May you always hit the right price for your spot  instances. Amen !

~Titan.

Eight years of writing

Yesterday, I opened my inbox to a pleasant shock and surprise. I had received a note from the Google analytics service which said that my blog had seen its 10000th visitor. Looking back, Is it an achievement? Well, Yes and No. No - because I didn't expect that many people to read what I wrote (and I'm thankful to them all !). Yes - because, I see the transition of Titan, the kid who had been awed by his exposure to a virtual world to the Titan, who is now realizing the value, virtue and vices of his virtual entity. And this blog has been one of the catalysts. As we say in tamizh movies, some elderly gentleman from the bygone era should have definitely said this (and I have no clue who he is) "A man's writing is a reflection of himself ". Actually, it doesn't matter if it hasn't been said so. It is at least largely true with most people that I have seen and known.

I happened to re-read my initial posts in this blog, which was written back in 2005. So many things have changed since then, so many things haven't. The most obvious is the content of the blog in itself. The transition from the half-broken language and muddled usage of words to a more clean and non-arbitrary diction in my writing (I recommend taking a look at my initial posts. I can assure you a 30 min episode worth laughter in it). But more interestingly, I found a significant change in the clarity of thought and in the structure of what I write. I do think (hope) there still is a fair bit of humor in my writing, but now, I also see the link between the words and the thoughts behind them. I'm guessing, this also means I have changed over time as a person. Maybe, maybe not.

Some of the complaints that I had been receiving from my readers (Yes! they do exist and they do reach out  to me by email too!)  is with the mix of technical (I mean, why not ! I'm a technology person still!) and non-technical posts in my blog. Some of them have out-right mentioned to me that those posts were boring, long, completely out of the blue and terribly uninteresting. Guilty as charged - but those posts are as much a part of me as my other posts are. And I'm sure you all know what I mean.

My writing has gained me many friends. When I was a kid, I remember the good old Amar chitra katha comics having a section called "Pen Pals". I used to wonder what it means. And now, I know. Some of my blogger friends (since I came to know them from the blogs) have been known to me for almost 8 years now, and I haven't met quite a few of them in person at all. But we have read our write-ups so many times over the years that, I think, if we happened to meet in the future, our conversation is likely to begin with a simple "Hey, What's up?". 

A common impediment that keeps many from writing on a public forum is the fear of saying something silly  and friends, relatives or prospective employers seeing it. I have seen many articles that spread this notion of being restrained or distancing your virtual avatar from yourself is a must. And to this end, I ask - if you are silly and write something silly and a person who you never meet in your life reads it, how does it matter? And if you do happen to meet him in the future, wouldn't he then realize that you are silly? I have found great satisfaction in my writing, and I write because it has helped me evolve. It has taught me to retrospect and constantly become a better person. I know many of my friends who would have loved to write, but haven't yet done so due to the fear of being judged. To them, I say - do write. Not for me, not for others, but for your own joy.

If that gentleman from the tamizh movies of the bygone era happens to stop by now and gives a talk about the virtues of writing, I'd pick up a coffee, switch off my cell-phone and listen.

Wabash Wisdom

At a fortunate moment of indecision, I decided to go all the way to Udupi, a restaurant 50 miles away from my dear WestLala, lured by the prospect of gobbling idlis and vadas (yes, you read the verbs right). As an after-effect of the above decision, I was forced to go for a jog in order to fit myself into my T-shirts again. In a certain sense, I'm happy that I decided to jog, since jogging along the northern parts of Wabash Heritage trail was one of my to-do things at West Lafayette. Just in case you are wondering, yes, I'm still looking for the second item in that list.  But let that be. For, as a fall out of my expedition today, I learnt many a thing that I'm postulating here as the dozen pearls of jogging wisdom. 


I learnt 

• that Rajinikanth is right, as always. He once said "vazhkai oru vattam da" (roughly translates to 'circle of life')
• that it takes at least 7 miles of running for your stomach to feel normal after hogging 11 medu vadas.
• that less traversed under-the-bridge pathways are a popular destination for men with OAB, no matter what country you are in.
• that 'The Road not taken' is an inspired poetry is a lie. Robert Frost had once taken a stroll along the N. 9th Street, Lafayette.
• that all roads in West Lafayette lead to the Happy Hollow park.
• why India does not compete in track and field events - we are just not meant to be athletes.
• It is final.  The winner in the category "best music track for inspired running" is Bhag bhag DK Bose. Sanjay Subrahmanyan's Upacaharamu cheseva comes a distant second.
• that fitness Apps in iPhone are not to be trusted for predicting distances. Ever. 
• what Jenny meant when she said "Run Forrest, run."
• that it is a good idea to take bath after running 9 miles . Apparently you turn into an effective rodent repellent after 7 miles.
• deer in west lafayette is not a fable. I saw two by the US-52.
• the meaning of life

Glossary:

WestLala - a rhyming pet name for West Lafayette, influenced by hours of watching innumerable tamizh songs with completely worthless lyrics.  
Bhag bhag DK Bose - hindi equivalent for encouraging a certain gentleman who goes by the name of D.K.Bose, to run. Really. Even Mr.Aamir Khan said so.
Upacaharamu cheseva - a carnatic composition by Saint Thyagaraja in the raga Bhairavi.
Rajinikanth - "      " intentionally left blank, undefined - the inventors of Boson particle at CERN are still working on this
Medu vadas - fried salted spiced lentil donuts. Salt - no sweet,.
US-52 - a very popular and landmark destination connecting the historic towns of Lafayette and West Lafayette across the might Wabash. All visitors of the Walmart definitely pass by this historic location.

A note to the posterity - this is the route that I ran today. May you find peace and prosperity and enough oil to light your lamps!

 
View Larger Map