Detecting a Memory Leak/Bloat of a Huge Code Base

Few weeks ago, I was maintaining a product with big traffic when I noticed that Nginx processes were eating up the server memory on random occasions. Nothing consistent, and the exact cause is totally unknown.

I initially thought this might be a memory leak, and I started doing memory profiling locally on my machine. I had no luck doing so, for several reasons, amongst of them: Huge code base and the testing environment was nothing like the production one in terms of data size and user base.

Then I thought why not treat the running process as a black box and try doing some linux stuff, so I logged the memory stats using this command:

Which dumps the size of the current request memory. It was good enough, but I used a better wrapper that helped me test on several platforms, using NewRelic code:

I waited for few hours and watched the memory consumed by the process after each request. Then I found out that I had a memory bloat, where the code was loading thousands of records due to wrong query.

It was a fun experiment, finding which request was eating the memory was a tedious task, but treating it as a black box turned out to be the right way to inspect it.

The Configurator Developer

I have been working recently on a little project where I had to hook together several 3rd party libs. Most of the time spent was about configuring these libs to make them work properly. Nothing fancy.

While working, it came to my realization that I have been working on similar projects recently, where most of the time is about gluing 3rd party stuff, and technical challenges are minimal.


The Configurator


For me, a developer who spends most of his working time gluing(configuring) 3rd party libs, is called a Configurator.

Now, don’t get me wrong, I’m not here to judge the nature of your work, since all of us have to do such type of configuring, maybe to try new things, quickly prototype something, or simply because we don’t have to reinvent the wheel. However, if most of your working time is going on configuring, then I do suggest you are not learning enough now. Actually, a question to ask yourself is:

What’s my competitive edge since anyone can do the same? Doctor Configuration?!!!

In such cases, I highly recommend that you dive deep from time to time in these 3rd party libs. Try to understand the code and the software design choices behind them. Try to fork and submit new new features.

Also, find time to start your own open source project. I have found that sharing code with others is the best way to become a better developer.

Bottom line, you need to realize whether you are in the configurator mode or not, and for how long. Based on that you can plan alternative paths.

What is it like to be a successful technical leader?


Being a successful technical leader is not limited to your technical ability. Your success exceeds your knowledge to your leadership skills.

Now, that you have a team to manage, work with and defend on, you need to be aware of the techniques that will soften everybody’s life, and of course make your career a successful one. This is something that’s harder than you might think as a start. There is a lot to learn and several skills to master.

I have gathered a collection of resources for you to read. I highly recommend that you read them and apply them occasionally. Best if luck!




Very Recommended Resources

Two harmful mistakes every developer should avoid

There are several traits of a successful employee every hiring manager seeks for. There are as well several mistakes a developer should really avoid. Those mistakes usually cause much trouble!
When I set to hire someone, I tell him explicitly that I can’t tolerate 2 of those unless agreed upon during the sprint/iteration life cycle.



A Deadline is a major thing, I like to call it a : holy date. Many developers don’t get the importance of deadlines unfortunately. However it’s simple: all teams depend on this date, business, marketing, pm and finance teams. When you have a problem setting this date, you simply screw everyone’s life, starting by shifting business plans and then adjusting other plans accordingly.

That’s why I ask new employees to take their time in giving estimations. If you need time to research something, then fine: ask for it and take it.

Use Gantt charts to do estimations. A task time shouldn’t exceed 4 hours, otherwise you have to split the task to minor ones.

Add time margin to your estimation. Some developers tend to be optimistic. Certain problems might arise. That’s why I ask all developers to add a margin of 2-3 days depending on the sprint/iteration length.

If something is blocking you then you need to raise it ASAP. If for some reason you can’t meet a deadline, then raise it ahead of time.


Code QualityCode quality is the most important factor in evaluating someone’s work. However, that’s not the reason of its importance. The importance of code quality comes from the old known fact; your code will be maintained by you or others in next iterations of the project.

If you deliver low quality code then, most probably the code will have to be rewritten after few iterations. Simply, you’ll waste months of planning and work.

However, code quality has no clear standards in terms of expectations. This is something that you need to consider and revisit every while and then. I ask team leads to set those standards in both design and implementation. Personally, I like both the SFEMS and the SOLID principles.

Pair programming is by far, the most reliable option for code quality. Testing is another one with a little cost of time at the beginning of practicing.

Both deadlines and code quality are so much important factors that help shapes the ninga of you. Please work on optimizing those skills. Stick to your team lead and take his advice when needed. It’s all about experience after all!

Joining N2V

Hello Dear Readers,

It’s my pleasure to announce that I have joined the marketing team of National Net Ventures — N2V , Amman branch. Therefore, I’ll relocate to Amman as well.
I’m excited to work with such a giant company helping entrepreneurs and fostering Internet industry in MENA region. Actually, that’s a part of my message in life.

I’ll give a hand in the technical efforts required by the marketing team to further expand the company’s work and message. Let’s see how things go 😛

Wish me good luck people 🙂