As an engineer working for large organizations, my goal is always to work on projects I can be proud of and learn a lot from while also benefiting the organization and its customers. If you are lucky, you’ll meet a project that checks all those boxes but if not, the best way to work on one is actually propose one by yourself. I have proposed and worked on a lot of my own projects and seen a lot of other people do it successfully. Someone recently asked me about how to find ideas to pitch your own projects and I verbally drew the following diagram for them. That actually came out better than I expected so I thought I should draw it for a post. Lot of the things in the diagram are self explanatory but I have added some thoughts for some aspects closer to my heart.

Now to be clear, proposing a project in a large organization goes way beyond just coming up with a compelling idea. Depending on where you work, you will have to come up with a case on why it will be impactful, aligns well with the product roadmap and actually feasible for implementation with resources that you are asking for. Even if everything aligns, you may not always get the resources if there are competing priorities and other internal factors that we don’t have to get into here :-)

This post however is only about idea generation as that’s the first step. The goal is to come up with something the leadership will be willing to fund and will benefit your internal and/or external customers while also maximizing your own learning. In my diagram, things that intersect with the orange oval typically tend to maximize the learning part (which can be a little subjective: not everyone finds the same things “cool”).

My personal favorite place to find ideas is the orange intersection where you find opportunities to improve the existing product. This often comes in form of optimizations to existing products. Many of my ideas fall under this area with applying load balancing optimizations for Lambda that a lot of my other posts talk about. These projects typically require you to deeply understand your existing product and also read a lot of research for what is possible in the area. Remember that a lot of research is not practical (often not even reproducible!) and there’s definitely such a thing as premature optimization. So in a way, optimization is a waiting game. You have to know about the inefficiencies in your products and have to track them with metrics for a long time before you realize it’s time to optimize stuff.

Premature optimization in my mind is optimization that pigeonholes your system into a state of lower flexibility which helps you really optimize things a lot (e.g. fixing your OS/architecture and building assembly hacks to make things run faster or with fewer resources). However, if you are an early stage product that hasn’t really narrowed down your niche, this type of optimization might make it harder for you to pivot or repurpose your system for other things. A simple system built using a bunch of popular open source products thrown together might be a much better fit in such situations even if the performance is suboptimal.

Another interesting aspect when coming up with these ideas is to attract the best engineers to work on them. When you propose your idea and get “funding”, that’s just someone saying it’s okay for 4 people to work on this for two quarters. You still need to find four people that you consider very good in various ways — great to brainstorm with, great programmers, experts with CI/CD tools etc. so there’s a marketing to engineers aspect to all of this. Some people find this easier than pitching to leadership while some find it harder. Apart from being considered a generally pleasant person to work with, the key aspect to this marketing in my opinion is just having a good handle on what folks find interesting and what they care about. It’s important to communicate why you are excited about the project but also what it will do for their skills, career development etc.