in Career

What I learned when I brought 99acres down with a stupid piece of code?

This is a story of an incident which happened in 2007 while I was working with InfoEdge in Noida. I regard this experience as one of most humbling experiences of my professional career. After this incident, I realised that anyone can make mistakes, and stupid ones at that. I also learned the importance of accepting your mistake and the consequences that come with it. I realized that accepting your mistakes is the only way to leave it behind in the past and move ahead towards the future.

At that time, I had been working with 99acres for more than two years and was a senior member of the technical team. I was implementing a small module on the homepage and search results page, the most important pages of the website. I was always very good at programming, and am very confident (sometimes over-confident too) about my programming skills too. I coded that module successfully, it was tested and Abhinav (my team leader) made it live on production servers. This was during the late evening time. After that, the load on the server was very high and abnormal, but I just left without giving it another thought which was so wrong, both as a member of the team and also knowing the fact that my code has just gone live on the site.

The next day, around 10 am the website stopped responding. Nobody knew what had happened, we could not even get a remote SSH connection to the servers. We had to ask Vivek, who has heading technology for 99acres and was the business head of Jeevansathi.com, to call the service providers in the US to do a manual reboot of the server. Once that was done, the site was back up again. But in another 2-3 hours, the same thing happened again. The service providers had informed us that the CPU and memory resources were getting depleted very fast. We thought it could be a hardware issue and asked the service providers to verify the same. Meanwhile the same cycle kept on repeating all throughout the day without any luck.

Lots of sweet and not so sweet memories between us two

Me & Abhinav in 2005

Abhinav asked me whether this could be because of my code, but I remember exactly what I had done and was sure that it could not have caused the problem. Even Abhinav and other members of the team checked the code and found no problem with it. It was only around the end of the day that I realised what was causing the resources to be exhausted. I modified a piece of code in one place in such a way that it being called from another place caused it to call itself recursively. And since this other place was the homepage, it was getting recursively called and quickly taking all the server resources resulting in the server crash. There was nothing wrong in the code, and this could have happened with anybody’s code, but my mistake was that I was so confident that I did not even bother to look into my code to check if anything could have gone wrong.

Once I realized this, I fixed the part of code that was causing the problem, and Abhinav quickly made it live. But as this had become a major issue, we had to let Vivek know what was the problem. I asked Abhinav what should I do, and he asked me to tell Vivek the truth and accept my mistake. I went to Vivek’s cabin, and informed him what was the issue and that it has been fixed. He said that I was one of the senior most guys in the team and he didn’t expected it from me. I apologized and said that I will be careful from now on and it will not happen again. He further added to forget this issue now and get back to work.

I could have easily covered that issue up and let nobody knew what was the problem. But accepting this in front of Vivek gave me the freedom to see ahead rather than getting stuck with the issue. The point was that anybody and everybody of us will do mistakes (as we are all human) so it is better to accept it ourselves, learn the lesson from it and move ahead for the future. Now when I look back, I realize that this was one of those days when I learnt a lot. Accepting this mistake gave me the courage and strength to lead the 99acres team in the future and accept others mistakes as a natural thing which will happen. I learnt that it is very important to forget the mistakes, but always remember the lessons that come with it, and move ahead with life confidently!!!

And for those who were with 99acres, the module which I coded was the “Featured Projects” module which was placed on the Homepage and the Search Results Page 🙂

Write a Comment

Comment

  1. I guess i was der part of 99acres – when dis happened, job well done Sumit 🙂

  2. Ye post main copy paste kar ke apne blog mein likhne wala hoon – bas thode name changes hain . Sumit ki jagah Kunal , Abhinav ki jagah Sumit and Vivek ki jagah Alok 🙂