Data Engineer

Key Responsibilities

• Design, develop, and support batch and incremental data ingestion pipelines into the data lake.
• Develop and optimize complex SQL queries, data transformations, and ETL workflows.
• Build, maintain, and monitor Airflow DAGs for orchestration and workflow automation.
• Analyze and troubleshoot data quality, data reconciliation, and pipeline execution issues.
• Perform end-to-end data lineage analysis and trace data from source systems to downstream consumers.
• Debug data discrepancies by investigating source systems, transformation logic, and target datasets.
• Support data migration and modernization initiatives from legacy platforms to cloud-based data platforms.
• Work closely with business stakeholders, architects, and upstream/downstream teams to understand data requirements.
• Validate data completeness, accuracy, and consistency across multiple systems.
• Participate in production support activities and resolve data-related incidents within SLA timelines.
• Develop reusable data engineering components and recommend process improvements.

Required Skills

Strong Expertise

• SQL / PL/SQL
  • Complex joins, CTEs, analytical functions
  • Query optimization and performance tuning
  • Data modeling and database concepts
• BigQuery
  • Query development and optimization
  • Partitioning and clustering concepts
  • Data loading and transformation techniques
• ETL/Data Engineering
  • Data ingestion patterns (Batch and Incremental)
  • Data transformations and validations
  • Data quality checks and reconciliation

Good to Have

• Apache Airflow
  • Developing and troubleshooting DAGs
  • Scheduling, monitoring, and dependency management
• Python
  • Basic to intermediate scripting
  • Data processing and automation
• Cloud Platforms (Preferably GCP)
  • Cloud Storage
  • BigQuery
  • Composer/Airflow