Director of Data

Join TBAuctions as the Director of Data, where you will own the company's data platform and lead the Data Platform Engineering team. You will be responsible for the technical direction of the data platform, developing a roadmap, leading data ingestion and integration, implementing platform-wide governance, enabling ML & AI workloads, and developing the Data Platform Engineering team. You should have 8+ years of experience in Data Engineering or Data Platform roles, with strong knowledge of Apache Spark internals and experience with cloud platforms.

• Ownership of the company's data platform, ensuring data is ingested, processed, governed, and made available reliably and securely for analytics.
• Leading the Data Platform Engineering team, setting engineering standards, coding practices, and delivery priorities, and being accountable for the team's growth and the platform's operational excellence.
• Developing a roadmap that balances business priorities with long-term platform investments, and making architectural decisions that keep the platform scalable, secure, and maintainable.
• You partner naturally with Product, Engineering, Machine Learning, Architecture, Security and business stakeholders, and you bring the same pragmatism to picking battles as you do to picking technology
• You think like an architect and work like an engineer: you can set a long-term technical direction and still get into the details of a Spark job or a pipeline that's misbehaving
• You've led engineering teams before, and you know how to turn a roadmap into shipped, reliable infrastructure
• Experience designing architectures that combine batch and streaming processing (e.g. Lambda or Kappa), including checkpointing, state management, exactly-once semantics and watermarking
• 8+ years in Data Engineering or Data Platform roles, including 3+ years leading engineering teams, with experience operating enterprise-scale data platforms in production
• Strong knowledge of Apache Spark internals (shuffling, driver vs. executor architecture, lazy evaluation) and experience building reliable, production-grade ELT/ETL pipelines
• Experience with CDC technologies such as Debezium
• Strong experience with Databricks or a comparable lakehouse platform, and with data modeling frameworks such as Kimball dimensional modeling
• Experience with data catalogs, data masking techniques and data governance platforms (DataHub, Atlan, OpenMetadata)
• Experience with cloud platforms (Azure preferred; AWS or GCP acceptable) and Infrastructure as Code (e.g. Terraform) and CI/CD pipelines
• Experience implementing Medallion Architecture using a transformation tool such as dbt
• Prometheus & Grafana
• Table formats: Delta Lake, Apache Iceberg
• Databricks, Apache Spark (Spark SQL, PySpark), Apache Kafka, Apache Airflow
• Kubernetes — deploying and maintaining stateful services (e.g. OpenMetadata)
• Git and DevOps practices (CI/CD)
• Technical skills: