Who are you?

The answer is some name and a story. Every society invents a story of themselves and their enemies. I say invent because Big Bang and Dinosaurs appear in no one, though we have enough evidence to prove that both of them happened. My best guess is that life is a virus, some core fragment of DNA which mutates giving rise to species. Sex works like a checksum to ensure that mutations can be recombined back to the core, resulting in birth. Death is main mechanism to ensure that evolutionary mistakes are not long lasting. Of course that is not the kind of story we are talking about. We are talking about the story of God and how he created men and women and what to do with neighbours etc. Something straight out of children’s story book and not a combination of philosophy, computer science and genetics. 

Let's be clear that all stories are false. They are precisely false because no one knows what the hell are we doing on this planet in the middle of no where, who created us or why.  But stories are good. Once enough people agree to a story, it becomes truth. Sometimes it takes more than a story to make it a truth. It needs monuments like temples and places of worship, things people can see around them. Sometimes it takes posters, flags and little symbols which people can keep in their homes. It also needs fashion statements, like beard or turban or cap or thread or necklace or uniform or something like that, so that you can carry your stories with you, everywhere you go. It takes festivals to remind to yourself and others of your existence and your difference and your special story. It needs loud music and songs to make it easy for people to remember the story and to force others to live with its noise. Being able to make money out of your stories cannot be understated. Money helps in giving livelihood to people whose sole job is story repetition and propagation.  The recent tools include WhatsApp messages, Facebook, ringtones, caller tunes, Display Picture, back glass of the car, door, etc. Story exists on its exposure, in minds of other people. Sometimes it takes violence, to kill others to assert your story over other stories, to teach others a lesson on not to mess with your story. The more other people need to adjust their lives to your story, the stronger your story becomes. Stories need investment and time and faith in face of difficulty. Story can only give so much power to you as much you are willing to invest in it. 

One precondition for the story is that it should include creation. Don’t worry about its stupidity. Every story demands faith and something is better than nothing. Who gives a damn about the specifics of the story? It is just means to an end. The stupider the story, the more faith it requires, the stronger it binds the people. The reality is unbelievable and uninteresting. The more you try to discover, more you get lost. The healing power of stories comes from faith not believability. It is a container, a placeholder. Here is where the buck stops. Social animal knows his own inadequacy and feels safe in the herd. We are the figment of our own imagination. We are a story, invented or borrowed or inherited. When many people share a story, it is called identity. Identity maintenance needs reinforcement, reciprocation, replication. It needs real hard work. 

Here are some ideas:

• Sun was horny and he wanted to marry the most beautiful princess in the whole world. King knew about this and kept the princess in a cellar, away from Sun. She used to come out only at night. The god of clouds was friend of Sun and he wanted to help him. One evening, he cluttered the sky with so much clouds that the king thought it is already night. When the princess came out, the clouds vanished letting the Sun take a look at her and she got pregnant by his sight. Knowing what has happened, the king threw the princess out of the kingdom. The son of the princess started a new kingdom. The rulers of that kingdom were called X. 
• The dark clouds had gathered and world was becoming a dangerous place to live. The wise men knew that something wrong was about to happen. The enemy was at the gate. They looked at the sky and heard the God say “Bring the water of seven rivers in one bucket and make the strongest bull drink it”. So they did as they were told and when the water was finished, instead of a bull they saw a warrior in its place. The great warrior defeated the enemy. The sons of that warrior are called X.
• I especially like the creation story which works through declaration. God was upset with the way the world has become and it was not going according to the plan. Then God said, let there be X and X was created from the earth and fire and God himself put breath it him. Then he told X to save the world, create a temple, give the offering and spread his message.  
• There was a King who liked to hunt elephants. One time he went to jungle with his army to hunt and found an elephant which was really huge and strong. He wanted to kill it. As the army came near, the elephant killed all his army. Then he looked at the King and was going to kill him, when the King bent down own his knees and asked for forgiveness. The elephant had mercy on the King and let him live. King requested the elephant to come and live with him in his palace. When the news of King coming to the palace with a big elephant was heard by the people, they started to worship him, thinking King has subdued the elephant. As soon as the elephant entered the palace, the King had him poisoned. It didn’t kill the elephant but made him sleep. The King tried to kill him but his skin was so strong that no weapon would cut him. Finally the King had him chained and put him in one big jail. The Kings glory started to spread as the news of his capture of a dangerous elephant. He organised a festival where people from his kingdom come and throw stones at the  elephant. The elephant undisturbed kept sleeping. One day it rained so much that the elephants jail was filled with water. He suffocated and woke up. Finding himself in chains, he easily broke them. He realised what the King has done and started killing everyone. He killed everyone in the palace, then he went to villages and then to other cities and in no time the whole kingdom was reduced to corpses. The God realising that someone is going to kill all of the mankind, appeared before the elephant. He turned the elephant into human and gave him the power of forgiveness. With the power of forgiveness elephant gave life back to everyone except the King and became the King himself. Thus started the reign of X. 

We also need a name but that is probably easier problem to deal with. Stories are the stuff that countries and religions are made of and so are people. They are just the necessary evil of social existence. Borrow one or make one, because if you don’t someone else will write your story and it may not be the one you like. 

Economics of Identity Politics

Before we go into the details of Identity Politics, we need to spend some in understanding the concept of life time value of a customer. Let's say a toothpaste tube lasts one month. Let's assume that X  is 20 years old and his expected life expectancy is 70 years. If the price of toothpaste is Rs 100, the life time value of X for the toothpaste manufacturer is (70-20) * 12  * 100 = Rs 60,000. 

Let's take another example: Let's say X’s kid goes to some school and he is in grade 1. The yearly school fees is Rs 2 lakhs. The lifetime value of the kid to the school is 12 * 2 = Rs 24 lakhs. What this calculation enables is putting a cap on maximum money manufacturer or school can spend on acquiring the customer. They may not make money on the first few sale, but over the lifetime of the customer, they will still make money. 

Let's try to see how this applies to politics in a democracy. In theory, in a democracy political parties will come up with policies and then the voters will judge them on the basis of these policies. The party with better policies win. Democracy in theory is like economics, where the invisible hand of rational self interest will cause parties to come up policies that matter to most and the voters will choose the most representative government. Just like everything else, theory is not quite applicable in practice. Coming up with better policies and doing good work is just plain hard. Even harder is selling the idea of why their policies are “better”. Let's have better tax collection to make infrastructure better is a really bad idea. No one likes it and no one wants to pay more tax. It so happens that “better” is different for different people. The worst part of this strategy is that the party needs to compete in every election to get back the people who voted for them the last time. This is very expensive and uncertain politics. This is bad business. 

Identity policies is a far better strategy. Voters are not expected to change their identity, by definition of identity. Once the voter is sold using identity, voter is acquired for lifetime. This amortises the cost of voter acquisition over the life time of the voter. What it means is that if the voter decides to vote because of identity, he will continue to vote for the same party unless he invents a new identity or other parties claim the same identity. No amount of good policy from competitors can swing his vote. This is a gem of a strategy. Not only do party reduced the cost of voter acquisition, increased voter stickiness but at the same time got complete independence from voter judgements around policy.

I love this quote from Sin City where Senator Roark says  “Power don't come from a badge or a gun. Power comes from lying. Lying big and gettin' the whole damn world to play along with you. Once you've got everybody agreeing with what they know in their hearts ain't true, you've got 'em by the balls”.  Identity voting is a pact with the devil, sale of soul. Once sold on identity, any rationale against party policies would either insult the identity or the self. It looks almost impossible to beat the vote unit economics of this strategy and the freedom of action it gives to the party. It attacks democracy exactly at the point which it was not evolved enough to handle, namely tyranny of the majority. The only available counter strategy against identity politics is more identity politics. The problem of course is that what looses out in this game is the integrity of nation. You only win this game by destroying the stadium.

Counting Representation

Democracy is great and so is the idea of representative government. The point where it starts hurting is that majority wins. By definition, majority is representative of just majority.  Even majority representing majority is questionable. Majority does represents majority when we have just two parties. But when we have more the two, we can be easily sure that majority is not represented. For example, in a three party, highly contested elections 34% would count as winning majority and it excludes 66% (real majority).  Essentially, majority of votes don't translate to majority in representation.

Surprisingly, it is easy to fix. All we need to change is what we count. The basic idea is to add the dimension of time to representation. Instead of the crude approximation of majority, we can have complete representation for everyone over time.  As popular is management circles, you only get what you measure.

How do we measure representation? As we discussed earlier, votes don't translate to majority representation. Well some of the votes do translate to representation. The votes which are cast to the winning party. The votes that are not cast and the votes that are not cast to the winning party are useless. They don't count toward representation.

Without much ado, here are the rules of the game:

1. Votes which result in representation are "spent"
2. Votes that don't get representation are "not spent"
3. Votes that are not cast are "not spent"

In other words, votes can be stored and used across elections.  Each vote is worth 5 years of representation. They work just like currency as store of "political value".  For example: say we had a two party system with 100 people and say party A won the election with 51 votes. These 51 votes are counted as spent. Rest 49 people get to keep their unrepresented vote for later use. In the next elections, 49 people will have 2 votes each and 51 will have one vote each. It is easy to see that with 98 votes, these 49 people will we able to easily get their representation.  

Given 65 years of life expectancy and 18 years as voting age, people get to vote around 10 times in their life. If the number of political parties is less that or equal to 10, even some 10% of the population will get a chance to form a government during their life time. To make this work with even larger number of parties, we will need to either have more frequent elections or allow inheritance of "political value", just like property and money.  This makes it possible for any arbitrary group of people to eventually form a government, perhaps once in few hundred years. 

It is easy to see that this method of counting representation leads to representation of all people over time. It will probably help stop the madness around "winning" elections and all that goes into it. Why? Because if you win, you make it easy for others to win the next time. If you loose, you chance of success increase over time. 

From what I could reason about, the system has two equilibriums. One: If people are cleanly partitioned into groups, over time each group gets representation and world becomes fair in terms of representation. If the groups are greedy, they will choose policies which advance their own groups. Sadly, the same strategy will be used by each of the other groups. Sad, but fair. 

The second equilibrium, would be towards enlarging or growing the size of these groups. Essentially, if the two groups can sort out their differences and work towards what is common and important for them, we get one less group and hopefully policies which work towards welfare of all the people in this larger group. The recursive logic will bring us to some manageable number of groups or even just one.  We might even see groups getting split when they cannot reconcile their differences, but that still leaves the process fair, just and representative. 

No matter which way the wind blows, we can always be sure that everyone is getting represented over time. If nothing else, it reduces the cost of running political parties and hopefully that is the money which can be used for advancement of the country and providing public goods. 

Criticism is most welcome!  

On eventually consistent file listing

Introduction 

Cheap s3 storage comes with unusual cost: correctness.  One of the key problems while working with highly partitioned data stored in s3, is the problem of eventual consistency in file listing. What exactly is this problem and how can we think about mitigating its impact: we will discuss in this post.

File listing is a very common operations since the invention of files. Given a directory or path, it gives us the list of files under that path. Tons of lines of code written over file systems, depend on correctness of this operation.  Unfortunately, this assumption breaks when the listing is done on s3 or for that matter any blob store.

Deconstructing file listing 

One way to think about eventual consistency of file listing is to argue that we get a wrong answer. This is correct to some extent but not powerful enough to do something about it. To do something about it, we need to dig a bit deeper and understand the nature of this wrongness. I find it useful to characterise this in the following form:

• Ghost files 
• Conceived files 

Lets try to understand what they mean. Ghost files are files which are listed by the file listing operation but they have actually been deleted from the file system. This is a very common reason for job failures in spark and hive. They are called ghost for obvious reasons. Conceived files on the other hand are those files which actually exist, but were not returned by the listing API.  In the happy (immediately unhappy) path, eventual consistency causes jobs to fail, because further operations on ghost file keep failing, irrespective of the number of retries. In the unhappy (short term happy) path, we have data loss because of conceived file, because they are simply missing in the final result,  resulting in incorrect answers.  

Given these two constructs, we can argue that the wrongness of a listing operation will occur either because of Ghost files (files which shouldn't be present in the listing but are) and conceived files (files which should be present in the listing but are not there). We can now have separate solutions for dealing with detection and consequences of these two file classes.

Dealing with Ghost files

Ghost file are files which shouldn't have existed in the listing API to start with. Once they show up, they cause different problem depending on what operations we are doing with these files. Most common problem would be subsequent file not found errors. One way to deal with this is to do a fresh listing operations and do a set subtraction.
Let A be the result of listing operation at time t1
and B be the result of listing operation at time t2,
where t2 > t1.
Set A-B i.e the files which are in A but not in B, is essentially the list of currently known ghost files. Once detected, we can choose to deal with them in some form. One simple way is to ignore the failures caused by ghost files, because we know they should fail. The other option is to remove them from our task queue, because we know they are not part of the final solution set. We might need to iterate multiple times (say, till a fixed point) to find out all the ghost files.

Dealing with Conceived files

Conceived files are the files which didn't even show up.
Lets again consider that A be the result of listing operation at time t1
and B be the result of listing operation at time t2,
where t2 > t1.
Set B-A i.e the files which are in B but were not in A, is essentially the list of current known conceived files. These are files which we would have missed if we only do a single listing operation. Once detected, we can choose to deal with them in some form.  Handling of conceived files is relatively simple. We just need to add them to our working set. We might need to iterate multiple times (say, till a fixed point) to find out all the conceived files and treat them as if they were part of the original listing operation. 

It is tempting to say why not wait until the system has become consistent before starting the work. In theory it works, it practice we don't know how much time it will take. Starting with whatever information we can get from the listing API, we get a head start and can keep revising our task set depending upon what further listing API reveals. What we get through this approach is correctness but without introducing any performance penalties.

In conclusion, we can deal with eventual consistency in file listing operations by repeating the listing operation, detecting ghost and conceived files and modifying our work queues to take our new knowledge about the listing status into account.

Don't screw yourself

It is a good principle to practice is life. In other terms it would mean don't do something stupid, or something that is not good for you.  Here and now, this is easier to practice. Add time and space to it and we have no clue what it means.

One of the popular ways of screwing yourself involves passage of time. Smoking is a classical example. Not saving, eating unhealthy food, no exercise, not learning new things, the list is endless.

Self screwing is not the only option. We can screw ourselves through other also. One of the simplest ways to punch yourself in the face is to punch someone else. Or consider not respecting right of other people to join the traffic. When we block someone, they block someone else. Since roads are a inherently connected, these small steps helps in creating larger deadlocks.

The point I wanted to make is: we are interconnected and these interconnections make is difficult to view our actions in isolation. These connections connect us not only to one another now but also ourselves to our future. Screwing others is just another way to screw yourself...not now, not here..but someday and somewhere.