Greg Wittel - Principal Software Engineer at Proofpoint

December, 2020 linkedin

Tell us a little about your current role: where do you work, your title and generally the sort of work do you and your team do.

I am a Principal Software Engineer at Proofpoint. My team is one of several focused on maintaining threat detection efficacy. We create backend infrastructure that fits between customer footprints and internal threat detection systems. The internal threat detection systems are built by multiple teams including mine. Depending on the purpose, our detection budget ranges from milliseconds to minutes. A given threat may be common run of the mill spam, to narrowly targeted one of a kind malware.

Typically, I focus on leading design and implementation of 1-2 projects at a time. Additionally, I may be acting as an advisor on projects that others will lead. Projects I work on usually impact the majority of our customers and may span multiple teams or business units. Depending on the nature of the project I may be contributing as a subject matter expert (e.g. security), or in the form of general engineering practice (e.g. architecture).

Outside of core projects, my time is often spent:

• Working on hard problems or troubleshooting when others need additional assistance.
• Working with other high level engineers to tackle organizational engineering challenges.
• Guiding junior engineers.

What does a “normal” Staff-plus engineer do at your company? Does your role look that way or does it differ?

While it varies across different parts of the company most Staff-plus engineers are leading key initiatives (new or ongoing), or delving into early research/proof-of-concept phase work. They will be deeply involved in one or both of the design and implementation of projects. Typically these will involve multiple teams, and possibly multiple divisions within the company.

For projects with customer facing elements, typically engineers are working with product teams to help shape the solution and end to end plan. Purely backend projects are usually engineering lead, coordinating with other teams as needed. At implementation time, most Staff-plus engineers are hands on, though some delegate work to the respective teams. By choice, I work in the hands on category.

At a high level, my role looks similar to other Principals at the company. Due to my teams’ infrastructure role, we work with a variety of teams, as well as being involved in a more varied set of projects than other teams. Because I have worked in many areas within Proofpoint, I often am asked to delve into security, systems, and scaling related challenges in addition to core project work. While being pulled into many differing areas can be challenging, it is also extremely helpful in learning how we can best help the company and our customers.

How do you spend your time day-to-day?

Because I have both project lead and implementation responsibilities, I typically try to allocate time within a week across three core areas: hands on technical, meetings/project management, and learning. I try to schedule time in blocks in order to reduce context switching between these areas. In addition we have no meetings days for focus time within the team.

While I have intentionally kept my role with a hands on element, the project lead responsibilities do mean that I write less code than I did as a Staff or Senior Engineer. I try to keep good notes on design, status, and next steps. This helps me resume from interruptions as fast as I can.

I try to optimize time in meetings in a few ways. First by pushing what I can to async means – e-mail, wikis, tickets, etc. Sometimes a meeting request really just requires some Q&A and can be answered via a short chat, email, or wiki update. This helps defend my teams’ time, as well as provides a reference doc for key information. Coming into a meeting, I want to have a plan whether it is an agenda, or a set of questions I hope to answer in the meeting. This helps speed up meetings and helps ensure the meeting is productive.

I strive for a continuous learning mindset. At work this means doing what I can to learn about new technologies, research, and the security landscape. Sometimes this covers immediately relevant information, other times its just interesting to me as an engineer. This helps keep me current, as well as broadens the set of options I can bring to bear in a new project.

Where do you feel most impactful as a Staff-plus engineer? A specific story would be grand.

As a Staff-plus engineer I want to help improve everyone’s chance of success. In the end, as an engineer if I am not making things better for our customer (internal or external), then I have work to do. In some cases this comes in the form of improving productivity, in others it is by helping projects succeed.

One of the most rewarding things for me is to be able to take a large project from idea to production. Asking the right questions early on in a project can save substantial amounts of time later on, as well as ensure it is on the right track. As a Staff-plus engineer, I have a seat at the table earlier in a projects’ lifetime. This gives me the opportunity to influence or course correct a project. Examples of factors requiring a course correction might include: ill defined (but critical) requirements, missing particular architectural failure modes, and correctly planning for scale.

When we’re trying to design a project plan, we won’t always have enough data to drive early decision making. It takes experience to ask the right questions about the data, and interpret results correctly. Having worked in security for well over a decade, I am able to use my experience to ask the right questions. The more I can help others do this themselves, they will be better able to work autonomously and do well.

Can you think of anything you’ve done as a Staff-plus engineer that you weren’t able to or wouldn’t have done before reaching that title?

The biggest things are around scope of influence and timing of being brought into a problem. As a Principal engineer, I may be working on projects ranging in scale from small ones within my team, to large company level efforts across multiple teams and business units. At the same time, being a higher level engineer has meant getting a seat at the table during the early decision making process. In some cases it is to provide guidance in shaping an effort (including pushing back on requests). In other cases, it is to lead a project. Additionally, I am more able to contribute to addressing organizational issues such as: providing feedback on leadership, challenges facing engineering teams as a whole, developer productivity, etc.

Do you spend time advocating for technology, practice, process or architectural change? What’s something you’ve advocated for? Can you share a story of influencing your organization?

While I spend time in an advocacy role, it is a relatively small portion of my time. For an active advocacy role, I work with several other Principal level engineers to identify areas that could use improvement. We’re seeking to identify, understand, and address problems in any of the people, process, technology realms. Effecting organizational change is difficult – we must have buy in from teams and their management chain. When we’re asking for a change, we’re asking teams to take time away from their core deliverables – time that they may not feel able to spare.

Two examples come to mind:

• A project that is slow or difficult for developers to implement, test, or deploy changes.
• An unstable, but important service requiring frequent fire fighting.

The impact of both problems is similar – engineers spend time on the wrong thing. No one wins: product won’t get the features on time, customers will see outages, engineers will be unhappy. Often teams know their trouble spots, as advocates we can provide additional support. This may come in the form of technical assistance or nudges to the management chain. From there, we can delve into root causes and push for focus on the right areas covering short through long term plans.

In a passive advocacy role, I contribute in a few ways. First, people frequently come to me for advice. Sometimes it is for my particular expertise, in other cases for a general engineers’ take. Working as a sounding board for others gives me a chance to learn, push for better designs, as well as guide engineers to others who can offer better assistance. At a secondary level, I try to lead by example. In the course of my work I want to set a bar others to reach. In a project this may be in core technical implementation, as well as oft-neglected portions such as documentation.

How do you keep in touch with how things really work as you spend less time on hands-on development?

I’ve specifically kept as much hands on work as I can. Despite this, I do code substantially less than I once did. My teams’ integration role means understanding other groups is an important factor for our success. Because we have to work across multiple organizational boundaries, I must have a very wide understanding of the end to end architecture across multiple teams and business units.

A secondary factor to keep in touch with how things work is by following my curiosity. I follow a continuous learning mindset – design docs, research papers, technical how-tos, etc. all play a role in this process. I read as much as possible, roughly grouping things in now/soon/whenever buckets. This helps me understand the landscape (in and out of company), as well as have a broader set of options that may be useful when future problems arise.

Ideally I should be able to find something to learn from any engineer. I cannot know everything, other engineers with differing experience have knowledge that can be shared. In talking with others, understanding their work is important. That may expose common problems, or give rise to new ideas. On more than one occasion, these sorts of discussions have resulted the creation of new customer facing products or features for Proofpoint.

How have you sponsored other engineers? Is sponsoring other engineers an important aspect of your role?

While I have sponsored other engineers, it is not a key aspect of my role. Usually sponsorship has meant recommending a junior engineer for a growth task, offering guidance as they progress. I am mindful of the fact that as someone who started early at the company I had the benefit of growing as an engineer as the company grew. Because we were small, I was able to have an out sized impact relative to my years of experience. I was also afforded the luxury of making mistakes as the customer impact was sometimes smaller. Now that we’re much larger, outages are very costly. This creates a tendency to be conservative, which in turn can impact opportunity for younger engineers. I want to find ways to create this opportunity for other engineers (as well as avoid my hard learned mistakes).

For sponsoring at a mentorship level, my bandwidth just doesn’t allow for me to do an effective job. In the past, I’ve mentored a few people with mixed results. My biggest failing is not being able to put in the required effort to make it a productive endeavor. Mentorship requires work on both sides; both in finding a fit (what type of help are they seeking, can you provide it?), as well as a way to ensure progress. Expecting to be able to do this in an ad-hoc manner limited the usefulness of the mentoring. Doing a better job requires careful forethought to ask mentees the right questions and have actionable measurable feedback. In time, I hope to be able to do more of this.

You first got the title Staff engineer at your current company. What was the process of getting promoted to Staff?

I first was promoted to the title of Staff Engineer at my current company. While there was no formal process, it started by asking questions to management around what it meant to grow: What should I be doing that I am not to be Staff (or Principal)? How or where can I improve?

It took a few years from when I started asking until my promotion to Staff. Work covered several aspects focusing on:

• A continued track record of important contributions to key projects.
• Constantly working across teams.
• Pushing the envelope in terms of tech, and delivery.

The main shift was being pulled into new, larger projects giving me both exposure as well as experience in new areas. Later on, I continued on this growth trajectory by working on larger efforts leading to later being promoted to a Principal Engineer.

What two or three factors were most important in you reaching Staff? How have the companies you joined, your location, or your education impacted your path?

Before reaching Staff at Proofpoint, I had switched to working remotely for several years (being in HQ prior to that). At the time, we had almost no remote engineers making me the odd one out. While I was incredibly productive, it hampered my career progression as I was increasingly disconnected from other groups. Once I returned to the office, it took time to rebuild those connections as the company was continually growing. Being present vs remote gave me the opportunity to interact with a broader set of people in the department. This gave rise to opportunity which would have been otherwise absent had I continued being a remote worker.

Technical skillset aside, strong communication (especially writing) was important as I needed to share the same information in varying ways to multiple audiences. This helped me build a deep understanding of our problem space, and my teams’ customers (usually other engineers or threat researchers). It also helped me refine the ability to share technical information at an audience appropriate level.

I had a fairly traditional Computer Science education, studying in many areas but ultimately focusing on computer security. As a graduate student, I ended up working on a new project area for my advisor. This required me to start a new large project from scratch by myself. Learning the cycle of research, design, implement, refine has been a critical skill through my career. One of the most challenging aspects at the time was in trimming my ambition (i.e. adjusting plans for your deadlines). After months of research, I had come to the realization that the scope was too large and needed to adjust my approach. It was a difficult, but humbling decision.

Learning to dig is another core skill – being able to dive into unfamiliar areas, pick apart, and reason through problems. Following curiosity, and being able to tackle problems in this manner is incredibly useful.

There is a popular idea that becoming a Staff engineer requires completing a “Staff Project.” Did you have a Staff Project, and if so what was it?

I did not have a Staff project. It helped that I started when Proofpoint was much smaller, smaller teams were able to have out sized impact. When I made Staff, we were launching several key efforts requiring careful coordination across multiple teams. My experience let me make high impact contributions in many places, as well as dive into new areas and get up to speed quickly.

Can you remember any piece of advice on reaching Staff that was particularly helpful for you?

I don’t remember any particular advice I received on reaching Staff. Over time there have been portions of advice now and again that stand out. One in particular has to do with messaging – while a blunt technical assessment may be fine for some audiences, it is not for others. While I try to be as straightforward as possible, sometimes that can lead to unintended consequences. For example, describing a quick patch as “throwaway” work may be accurate, someone might interpret that as saying their work has no value. In providing opinions, I try to be more thoughtful about how and what I communicate to avoid such situations.

What about a piece of advice for someone who has just started as a Staff engineer?

At an individual level, the requirements for Staff-plus may vary. To that end, work to understand expectations from your manager and the broader organization. In reaching Staff, you’ve likely shown that you are already able to perform at the expected level. In making it official, expectations of others may change as you cement your position and reach toward the next level.

Typically these involve the increasing importance of soft skills. Communication skills (both verbal and written) become increasingly critical. Being able to share knowledge and adjust your message for the audience (e.g. up and down level, more or less technical) is vital. In other words, in order to get your message across, know your audience and adjust accordingly. There will be more demands on your time. This will likely mean a schedule with fewer large blocks of time where you can do deep technical work. In order to compensate, develop strategies to maximize the time you do have, as well as minimize recovery time from switching tasks. Part of this effort may mean allowing others to take on tasks you might have previously handled. In other cases, it may also mean saying ’no'.

As you work on leading larger projects, building teams and improving engineers is another factor. In some cases, this means leading by example (doing). In others, invest effort in boosting other engineers – find and eliminate toil, teach, design and architect with productivity in mind. I naturally gravitated toward the idea of what is sometimes called product oriented engineering. Largely this boils down to knowing your customer; be it internal or external. In doing this you can cut through the noise – taking an ill defined problem, work to clarify, and come up with a reasonable solution. If you’re handed a solution, it will need careful evaluation and work with stakeholders to come to an agreement. One must ask, does the proposal actually solve the problem? Is it the right problem to be solving? This can lead to conflict, and navigating this can be tricky at times.

The last thing is humility. You may be a Staff-plus engineer, but you will never know everything or have time to do it all. Its OK to ask for help. Its OK to say “I don’t know”; even better to to add, “but I will find out.” Learn from failure and mistakes. It is exceedingly difficult to improve if you cannot admit a mistake, flaw, poor decision, etc. Having a blame free (post-mortem) problem solving mindset helps you and others focus on what went wrong and how to avoid it in the future.

Did you ever consider engineering management, and if so how did you decide to pursue the staff engineer path?

Yes, I have considered it from time to time. Personally, I am more happy to be focused on creating the product as an engineer, rather than creating teams to build the product. The most effective managers I’ve had over the years dedicate a lot of time to things that I am not excited about doing – meetings, managing expectations (up and down), politics, filtering noise, recruiting, etc. While at a Staff-plus level, you do handle more of these things it is not a primary duty.

The tradeoff is that as an engineer, I can ask for resources, while a manager can allocate resources. At times this can be frustrating, but given the other core aspects of a managers’ job, I prefer staying on the technical track.

What are some resources (books, blogs, people, etc) you’ve learned from? Who are your role models in the field?

With respect to engineering I follow a mix of technical, and non-technical topics. Technical writing is an under-taught and under utilized skillset. Some of my favorite technical writers include:

• Julia Evans - Her ability to introduce technical topics into approachable writeups is particularly wonderful.
• Salvatore Sanfilippo (aka antirez) - His deep dive into internals, as well as the coding practices with respect to readability are concepts I take to heart.
• Steve Klabnik - Known for his Rust advocacy, his posts are highly readable and an example for others to follow.
• Edward Tufte - His work on presenting data is insightful and illuminating.

There are a lot of great deep technical blogs out there and its down to your area of interest. Recently I’ve been spending more time on performance analysis, and distributed systems. Some of my favorite writers in this area include Martin Thompson, Richard Startin, and aphyr.

On non-technical track I am fond of work including Rands in Repose, and Camille Fournier’s The Manager’s Path.

Next chapter: John Zittlau - Principal Engineer at Jobber

Previous chapter: Stephen Wan - Staff Engineer at Samsara

Book

If you've enjoyed reading the stories and guides on staffeng.com, you might also enjoy Staff Engineer: Leadership beyond the management track, which features many of these guides and stories.