Lead Data Engineer

Join Two Circles, a leading sports data and analytics company, as a Lead Data Engineer. In this role, you will be a key part of our Partnership Intelligence > Measurement Team, specializing in the ingestion, transformation, and storage of large-scale social and broadcast data. You will design and implement data pipelines, support the ongoing maintenance and operations of the data platform, and contribute to our DevOps culture. Ideal candidates will have 5+ years of experience in software engineering with a data focus, proficiency in Python and C#, and experience with cloud infrastructure and services.

• Design and implement data pipelines with a clear focus on data quality and reliability, ensuring the efficient ingestion, transformation, and storage of large-scale social and broadcast data.
• Act as an advocate for development best practices, including technical design reviews, implementing test plans, test-driven development, monitoring/alerting, peer code reviews, and documentation.
• Collaborate with cross-functional product teams to deliver a high-quality SaaS-based data platform, partnering with both Product Management and stakeholders to deliver the vision, roadmap, and data strategy.
• Experience designing and managing large scale data pipelines is a must for this role, as well as a keen interest to explore new technologies and solutions
• We are looking for someone who brings experience and passion working with the flow of data and infrastructure
• 5+ years of experience writing, debugging and refining database queries
• 5+ years of experience building or maintaining cloud infrastructure and services (preferably AWS)
• 5+ years of experience with object oriented languages (preferably Python and C# .Net)
• 8+ years of experience in software engineering with a data focus, working in a fast-paced, agile environment
• 3+ years experience automating repetitive tasks (like setups, developer onboarding, support, moving data, etc.)
• Proficiency in Python and C#. Both are required to be successful in this role
• Familiarity with monitoring tools for data pipeline performance such as New Relic
• Strong experience with containerization such as Docker
• Experience with writing unit tests using frameworks such as PyTest for quality assurance
• Experience developing against APIs and building scalable systems
• Understanding architectural differences between streaming and batch-oriented integrations
• Proficiency in Snowflake and MS SQL
• Experience with various orchestration and batch processing technologies like Apache Airflow