What I learned when I brought 99acres down with a stupid piece of code?

This is a story of an incident which happened in 2007 while I was working with InfoEdge in Noida. I regard this experience as one of most humbling experiences of my professional career. After this incident, I realised that anyone can make mistakes, and stupid ones at that. I also learned the importance of accepting your mistake and the consequences that come with it. I realized that accepting your mistakes is the only way to leave it behind in the past and move ahead towards the future.

At that time, I had been working with 99acres for more than two years and was a senior member of the technical team. I was implementing a small module on the homepage and search results page, the most important pages of the website. I was always very good at programming, and am very confident (sometimes over-confident too) about my programming skills too. I coded that module successfully, it was tested and Abhinav (my team leader) made it live on production servers. This was during the late evening time. After that, the load on the server was very high and abnormal, but I just left without giving it another thought which was so wrong, both as a member of the team and also knowing the fact that my code has just gone live on the site.

The next day, around 10 am the website stopped responding. Nobody knew what had happened, we could not even get a remote SSH connection to the servers. We had to ask Vivek, who has heading technology for 99acres and was the business head of Jeevansathi.com, to call the service providers in the US to do a manual reboot of the server. Once that was done, the site was back up again. But in another 2-3 hours, the same thing happened again. The service providers had informed us that the CPU and memory resources were getting depleted very fast. We thought it could be a hardware issue and asked the service providers to verify the same. Meanwhile the same cycle kept on repeating all throughout the day without any luck.

Lots of sweet and not so sweet memories between us two

Me & Abhinav in 2005

Abhinav asked me whether this could be because of my code, but I remember exactly what I had done and was sure that it could not have caused the problem. Even Abhinav and other members of the team checked the code and found no problem with it. It was only around the end of the day that I realised what was causing the resources to be exhausted. I modified a piece of code in one place in such a way that it being called from another place caused it to call itself recursively. And since this other place was the homepage, it was getting recursively called and quickly taking all the server resources resulting in the server crash. There was nothing wrong in the code, and this could have happened with anybody’s code, but my mistake was that I was so confident that I did not even bother to look into my code to check if anything could have gone wrong.

Once I realized this, I fixed the part of code that was causing the problem, and Abhinav quickly made it live. But as this had become a major issue, we had to let Vivek know what was the problem. I asked Abhinav what should I do, and he asked me to tell Vivek the truth and accept my mistake. I went to Vivek’s cabin, and informed him what was the issue and that it has been fixed. He said that I was one of the senior most guys in the team and he didn’t expected it from me. I apologized and said that I will be careful from now on and it will not happen again. He further added to forget this issue now and get back to work.

I could have easily covered that issue up and let nobody knew what was the problem. But accepting this in front of Vivek gave me the freedom to see ahead rather than getting stuck with the issue. The point was that anybody and everybody of us will do mistakes (as we are all human) so it is better to accept it ourselves, learn the lesson from it and move ahead for the future. Now when I look back, I realize that this was one of those days when I learnt a lot. Accepting this mistake gave me the courage and strength to lead the 99acres team in the future and accept others mistakes as a natural thing which will happen. I learnt that it is very important to forget the mistakes, but always remember the lessons that come with it, and move ahead with life confidently!!!

And for those who were with 99acres, the module which I coded was the “Featured Projects” module which was placed on the Homepage and the Search Results Page 🙂

What I learned when Neha kept quiet for 3 days?

This post is about an incident which happened when I was leading the 99acres team while working with InfoEdge in Noida. This experience shook me and I still remember it very vividly. It made me realize the extra responsibilities and sensitivity which a leader must accept. I am a very fun loving and jovial person and some fun / jokes / mockery is a part of my daily life. But sometimes, you need to balance those keeping in mind the position you are in.

Among the 14 members in my team, Neha was one of the best resources I had. Always the go-to person in case I want something done urgently and efficiently. Having worked with her for quite some time and sitting right next to her, she was also a very good friend and we shared a very good working relationship. Maybe thats why I still remember this incident. One day when I came to office I realized that Neha was surprisingly quite. She was talking only formally, and her mischievous look was also missing from her face. I thought it might be some personal issue and let the day pass. But when she was the same quite person the next day. I pondered that could it be because of something I did or said, but I dismissed that thought as soon as it came in my head. I even asked her what happened but she said she was fine.

When she was uncharacteristically quite even on the third day, I started thinking of something I might have done to upset her. On the evening of the third day, I finally asked her whether her behavior for the last few days was because of something I did or said. And to my utter surprise, she said “YES“!! I could not believe that she was upset at something I said and did not mention it to me for the last three days. On further probing, she told me she has talked to her sister, and she advised her to talk to me directly about the issue. And the “issue” was that there was some incident which required urgent attention and I went to Neha with another person (from another team) to debug it. After Neha and myself figured out the issue and what would be required to sort it out, I made a comment “Tumhara response time bada slow hai. It was just a casual remark to tease her and nothing related to that incident or her performance. It was a part of the “mocking” nature that is very much a part of me.

As I said earlier, I often make sarcastic and mocking about people and situations for humor and this was one such incident. But coming from her Team Leader just after she had finished an urgent task, it was very unfair of me to having made that comment at “exactly the inappropriate moment“. No doubt she misinterpreted it as a comment on her performance and took it to heart. I know how anyone will feel if a work well done is criticized instead of the appreciation which one expects out of such a task. And the fact that she was upset for three days and never told me despite sitting next to me totally hit me like anything. And how unfair of me to even think about something that I might have done to upset her and then dismissing that thought in a blink? When she opened up to me, I cleared up with her soon after and we were back to normal terms and she was her mischievous self.

I learned two things from this incident. Firstly, as a leader, it should be my responsibility to make sure what comes out of my mouth does not hurt others (or is misinterpreted). And given the very casual and informal culture we had in the team, I should have realised that situation (with Neha) was not the right moment for a sarcastic remark, and that too in front of a third person. I learned to be more careful in future and keeping the right balance between fun and serious talk. It is very important that when I joke around, people should not take it a serious remark by their Team Leader, and similarly, when saying something serious, that should also never be regarded as a joke.

The second thing I learned from this incident was following up and making sure your point is clearly understood by the other party or not. I had the thought a few times that Neha might be upset of something I said or did but I never gave it a serious thought. Had I cleared up with her earlier, she might have opened up to me and would not have stayed upset for so long. It is very natural for us as human beings to misinterpret something said to us as something else. Since then, I always try to make sure to get my point across so that there are no wrong interpretations. The same holds true if I am the hearing party. Instead of assuming that “this means that“, I often restate the statement to clarify to the other side what I understood from his / her communication. This way, when I leave the table, I am sure we both are on the same page and no doubts and misinterpretations remain.

I would not say that I am not doing similar mistakes now, but I am more careful and cautious now when faced with similar situations. Keeping this balance between fun and serious stuff is very important. And I am getting better at it with time. Certainly from those times at InfoEdge.