Data Engineer / Lead Data Engineer

Responsibilities

• Pipeline development & data modeling
• -
• Build and maintain batch data pipelines that ingest from Workday (HCM, Finance, Student) and legacy source systems into the Bronze layer
• -
• Develop transformation logic across Bronze / Silver / Gold layers using Spark, Python, and SQL on Databricks
• -
• Model data for analytics consumption, balancing performance, reliability, and maintainability
• -
• Orchestrate and schedule workloads (e.g., Databricks Workflows, Azure Data Factory, or Fabric pipelines) and monitor them for reliability
• -
• Implement data-quality checks and validation within pipelines
• Platform & governance alignment
• -
• Work within the federated architecture and Unity Catalog security model, respecting the boundary between institution-administered and centrally managed catalogs
• -
• Follow the shared transformation standards and reference patterns set by the Solution Architect
• -
• Contribute to documentation, code reviews, and version-controlled deployment
• Lead-level responsibilities (Lead Data Engineer)
• -
• Serve as senior technical lead on complex integrations and cross-institution data workstreams
• -
• Define and enforce engineering standards, patterns, and best practices
• -
• Mentor and provide technical guidance and code review to other engineers
• -
• Partner with the Solution Architect and governance leads on design decisions and platform improvements
• The responsibilities listed above are representative of the role and may be adjusted to meet organizational priorities.

Qualifications & Requirements

• -
• Data Engineer: 3–6 years building production data pipelines. Lead Data Engineer: 7–10 years, with demonstrated technical leadership
• -
• Hands-on expertise with Apache Spark, Python, and SQL in a Databricks environment
• -
• Experience designing and building medallion or layered data architectures (Bronze / Silver / Gold or equivalent)
• -
• Proven experience integrating data from multiple ERP / SIS / operational source systems into an analytics platform
• -
• Experience with batch pipeline orchestration and scheduling (Databricks Workflows, Azure Data Factory, or similar)
• -
• Working understanding of data governance, security, and access concepts in a shared or multi-tenant environment
• -
• Strong problem-solving skills with a consistent focus on data quality and reliability
• -
• Excellent communication and collaboration skills on a remote team
• -
• Bachelor’s degree in a technical field or equivalent practical experience

Preferred Qualifications

• -
• Microsoft Fabric and/or Power BI experience
• -
• Data-quality tooling such as Great Expectations — expected for the Lead level
• -
• Experience with higher education source systems: Workday, Banner (Oracle), Colleague (SQL Server), Jenzabar, PowerFAIDS
• -
• Unity Catalog experience and familiarity with CI/CD for data (Git-based workflows, DevOps pipelines)
• -
• Exposure to streaming or near-real-time ingestion patterns
• -
• Databricks certification (e.g., Databricks Certified Data Engineer)