At Abnormal Security we pride ourselves on our excellent machine learning (ML) engineering team. Every day, our ML engineers solve problems that make our groundbreaking technology even more effective at stopping the most advanced socially-engineered attacks.
As the company grows and our team expands, we recognize our success depends on inviting the right people to join us on our mission to build the next generation of email security. This is why we have taken the time to define what effectiveness means at Abnormal.
Here, we share some patterns that we use to distinguish between effective and ineffective ML engineers.
- Opting for Simplicity
An ineffective ML engineer uses complex algorithms without justification. They are quick to throw away existing solutions they don’t fully understand, dismissing heuristics or manual transformations in favor of fancy models.
An effective ML engineer starts by thoroughly studying the problem and the data first. They like to use the simplest approach that can completely solve the problem. They fully understand the current stack before proposing new solutions and don’t increase complexity without a sound justification.
- Focusing on Impact
An ineffective ML engineer spends weeks on projects that have no clear path to production. They don’t think about the environments in which their models will run online, or how their work will impact the product.
An effective ML engineer thinks about the online launch from the beginning of each project. They chart the path from offline experimentation to product impact and design their offline test plan to derisk the biggest uncertainties.
- Embracing Versatility
An ineffective ML engineer does not understand the software systems they rely upon. They become blocked easily when the exact datasets or tools they need are not available and rely heavily on software teams to support them.
An effective ML engineer is a strong programmer. They are comfortable outside of notebooks and write well-tested and readable data pipelines. They can read and write serving code as well as deploy their own models and feature serving systems when necessary.
- Avoiding Assumptions
An ineffective ML engineer makes blind assumptions. They might assume that a feature is always accurate, that the labels they use for training represent ground truth, or that a mean is a good default for all of their continuous features. An ineffective ML engineer doesn’t question where their dataset came from, or how their model will interact with the other parts of the system.
An effective ML engineer always questions their assumptions. They have good instincts, but they aren’t afraid to challenge them. They ask questions about anything they are unsure about and write down the answers so they can reference them later. An effective ML engineer loves data wrangling—starting with the client-side signals that feed the model features and ending with the product metrics that respond to model performance.
- Prioritizing Reproducibility
An ineffective ML engineer views the trained model as the output of their work. They don’t prioritize cleaning, testing, or refactoring their data processing or model training code.
An effective ML engineer understands the importance of reproducibility and views reproducible training pipelines as the output of their work. They refactor and clean their training code as part of the modeling process, and they write runbooks that are detailed enough for a new hire to understand.
- Managing Uncertainty
An ineffective ML engineer is unwilling to make commitments for how long something will take, or how much of an improvement they will be able to make.
An effective ML engineer knows that ML projects have high uncertainty but also understands that a software organization cannot function without reliable estimates. They can decompose a problem and propose a path that accounts for multiple possible directions. An effective ML engineer is detail-oriented and mitigates risks early with sanity checks and simple tests.
- Overcoming Roadblocks
An ineffective ML engineer accepts constraints at face value. They struggle to understand why certain things are hard or to communicate the problems they face to their team and leadership.
An effective ML engineer questions every constraint until they fully understand it. They propose new solutions based on first principles, external resources, or experience.
- Designing for Resilience
An ineffective ML engineer doesn’t prioritize change resilience. Their models and features adapt poorly to infrastructure changes or data distribution shifts. They optimize for today’s performance at the expense of tomorrow’s generalizability.
An effective ML engineer produces generalizable systems. Their models and features don’t fall over in the face of distribution shifts like new customers, products, or platforms. They prefer simpler and more adaptable solutions.
Embodying Effectiveness as an ML Engineer
Effectiveness means different things in different organizations. An ML engineer in a small startup needs to prioritize versatility and velocity, whereas the most effective engineers at large companies need to build on the work of others to drive impact. Growing as an ML engineer requires understanding what effectiveness means in your organization.
