Senior Data Engineer

Job type: Permanent
Contact name: Darren Dom

Contact email: darren@ds-group.co.uk
Job ref: 32347
Published: 21 days ago
Startdate: 04/07/2022

Our client helps companies optimize prices and discounts using ML. Their optimizer tests different price points to study the impact on revenue and other downstream metrics. They segment users based on their characteristics such as their location and usage patterns to make sure a director in the US and a student in India aren't offered the same price or discount.

In less than 6 months the company has gone from an idea to a revenue-generating product used by household names in tech including Linktree, IFTTT and Skillshare, among others.

 

They are now looking to hire their first Senior Data Engineer to join a very strong technical team and build together a system that will help power the world's prices.

 

Team

• Technical founders. Their founders are ex-BlackRock quants and startup veterans focused on building cutting-edge products that people want.

• Small team of senior engineers. You will get to work with ex-founders, ex-Yandex, ex-Avito engineers and talented data-scientists. They made a conscious decision to keep the team small and exceptional.

• Scrappy. They are problem solvers at heart. their sales automation is being run from an old laptop under the CEO's couch. They like to accomplish as much as possible with as little resources as possible.

• International. No two employees at Corrily are in, or from, the same country - or even in the same timezone. They are in Taiwan, Russia, the UK, Slovenia, Spain, the US, ... The average Corrily employee speaks 3.5 languages. They're constantly traveling.

• Flat. They do not have hierarchies and their founders are strong believers in flat hierarchies and even almost launched a startup about it.

• Owners. Everyone at Corrily is an owner. Employees set their agendas and are expected to understand their clients and business and come up with creative solutions to meet their needs.

 

And they have cool benefits too!

• Generous equity package

• Generous compensation

• Fully remote worldwide

• Unlimited time off

• Flexible hours

• Healthcare where needed

 

Stack

Infrastructure: GCP fully handled via Terraform (K8s, BigTable, BigQuery, Cloud Run, Cloud Functions)

Databases: BigTable for everything latency-sensitive (They developed a library on top of it to make it easier to use), BigQuery for analytics tasks, Hasura w/ PostgreSQL back-end for configuration / user management

ML: Currently using multiple frameworks (PyTorch for tensor manipulation / legacy neural network models, Nevergrad for non-convex optimization, cvxpy for convex optimization, Pandas and Numpy

for data manipulation). The ML stack evolves very quickly as they develop new versions of their optimizer. Their mono repo is mostly written in Python but they are expecting to expand this in the near future 

 

Responsibilities

• Design ETL/ELT pipelines allowing for each client to provide their own definitions of concepts such as "active subscriber" or "MRR"

• Own data pipelines and infrastructure, optimizing for extensibility as opposed to performance

• Contribute to python data libraries to make it easy for clients to interact with their data

• Collaborate with data-scientists to make data accessible and provide valuable insights to clients

• Define and improve training pipelines

• Create checks and improve observability in data systems

 

Requirements

• Strong programming and algorithmic skills.

• Strong experience with data processing (such as DBT, Hadoop, Spark, Pig, Hive, MapReduce etc).

• Experience scoping ETL/ELT projects from day 0 all the way to completion

• Solid Python skills

 

We encourage you to apply even if you don't meet all the criteria. We are looking for the right folks and can help fill in the gaps!

 

During interviews we look for:

• Technical skills: A good candidate should be able to define and implement ETL/ELT pipelines from scratch with scale and maintainability at heart. Typically, candidates that we see perform well have in the past led personal or professional projects to completion. Open source development, curiosity in learning about new technologies, propensity to prefer practical solutions to hyped ones are all things we value.

• Proactivity: What makes the difference between a good candidate and a great one tends to be proactivity. Are you purely following tasks, or are you thinking through why you are doing things, thinking of better ways to go about it, and pushing things forward?

• Communication / Personality: Any hint of douchiness is a cause for rejection, regardless of technical skill. We strive to make sure we maintain a productive collaborative environment. We value candidates who can express complicated ideas simply and clearly.