Senior Data Engineer: Massive Graph Processing

  • BHO Tech
  • San Francisco, CA, USA
  • Jul 08, 2024
Full time Engineering

Job Description

Help us process a trillion edge graph as quickly and efficiently as possible. Were Identity Engineering, and we maintain a massive graph that connects together the different identifiers for consumers (e.g., anonymized email addresses and phone numbers) and the devices they use online. 

The engineering systems weve developed are constantly ingesting new edges from thousands of different sources and finding numerous types of relevant paths to power our suite of core products. 

You Will:

Get to work on projects such as:

  • Pregel Path Computer: This system finds relevant graph paths using the pregel graph computation framework as implemented in Apache Giraph. There are challenges in running Giraph at the scale of our graph and were constantly looking to refine our Pregel algorithms.
  • Edge Ingestion and Partitioning Framework: We could never process all trillion edges at once and luckily we dont have to. Instead we process subgraphs that contain specific types of edges. Our edge ingestion and partitioning framework manages different Hadoop datastores for different types of edges and automates the ingestion of new edge data. It leverages our Seek MSJ framework to efficiently incorporate new data into existing edge stores.
  • Path Computation as a Service: We provide a service to our other engineering teams for finding specific types of paths within our massive graph. It handles 20,000 requests a day and this is possible due to its use of caching and intelligently batching similar request together.

Your Team:

  • We take pride in operating as a high performance team, while maintaining kindness and humility.
  • We find feedback to be important in helping us grow as individuals and as a team; were always looking for chances to share positive and constructive feedback.
  • We develop in Java and use MapReduce, Giraph, and Spark. Were open to new technologies and languages if they help us better solve a problem.
  • We currently use a 79,800 core Hadoop cluster with 90 PB of disk space and 256 TB of RAM (shared across all of our data engineering) to power our systems. Were also exploring moving everything to AWS.

About You:

  • Have 3+ years of experience writing and deploying production code.
  • Have a passion for building large scale, distributed systems and are comfortable writing high performance code.
  • You love mentoring junior engineers, and deploying best practices.
  • Have a startup personality: smart, ethical, friendly, hard-working and productive.
  • Are a data enthusiast who wants to be surrounded by brilliant teammates and huge challenges.
  • Excellent communication and presentation skills.

Bonus Points:

  • People Leadership Experience


  • People. Work with talented, collaborative, and friendly people who love what they do.
  • Food. Enjoy catered meals, boundless snacks, and the occasional food truck.
  • Fun. We host events such as game nights, happy hours, camping trips, and sports leagues.
  • Stock. Every employee is a stakeholder in our future. Health and Saving. Receive the benefits of comprehensive health, dental, vision and disability insurance along with a 401k matching plan.
  • Location. Work in the heart of San Francisco and take advantage of our commuter benefits.

More about us:

We are the leader in data connectivity, helping the worlds largest brands use their data to improve customer interactions on any channel and device. We thrive on mind-bending technical challenges and value entrepreneurship, humility, and constant personal growth. 

There is so much more that we want to build and that we could continue to improve. We value strong engineers who are agile enough to hit the ground running and tackle challenges. 

To all recruitment agencies: We do not accept agency resumes. Please do not forward resumes to our jobs alias, our employees or any other company location. We are not responsible for any fees related to unsolicited resumes.

We are an affirmative action and equal opportunity employer (AA/EOE/W/M/Vet/Disabled) and does not discriminate in recruiting, hiring, training, promotion or other employment of associates or the awarding of subcontracts because of a person's race, color, sex, age, religion, national origin, protected veteran, disability, sexual orientation, gender identity, genetics or other protected status. Qualified applicants with arrest and conviction records will be considered for the position in accordance with the San Francisco Fair Chance Ordinance.

Best Regards,
Kris Young
Account Manager
BHO Tech
San Jose, San Francisco CA
Phone: 866 816-1615 x 823