GCP Data Engineer

1. Core Data Engineering Skills

Big Data Technologies: Proficiency in using GCP's big data tools like

• BigQuery: For data warehousing and SQL analytics.
• Dataproc: For running Spark and Hadoop clusters.
• Airflow: For pipeline Orchestration
• Dataflow: For stream and batch data processing.(High level Idea)
• Pub/Sub: For real-time messaging and event ingestion.(High level Idea)
• Data Modeling: Experience/Knowledge on designing scalable, efficient data models for OLAP and OLTP use cases.

ETL/ELT Pipelines

Expertise in building automated, scalable, and reliable pipelines using custom Python/Scala solutions or Cloud Data Fusion.

1. Programming and Scripting

• Strong coding skills in Python, SQL , and optionally Java/Scala.
• Familiarity with APIs and SDKs for GCP services to build custom data solutions.

1. Cloud Infrastructure

• Understanding of GCP services such as Cloud Storage, Compute Engine, and Cloud Functions.
• Familiarity with Kubernetes (GKE) and containerisation for deploying data pipelines. (Optional but Good to have)

1. DevOps and CI/CD

• Experience setting up CI/CD pipelines using Cloud Build, GitHub Actions, or other tools.
• Monitoring and logging tools like Cloud Monitoring and Cloud Logging for production workflows.
• Soft Skills: -

1. Innovation and Problem-Solving

• Ability to think creatively and design innovative solutions for complex data challenges.
• Experience in prototyping and experimenting with cutting-edge GCP tools or third-party integrations.
• Strong analytical mindset to transform raw data into actionable insights.

1. Collaboration

• Teamwork: Ability to collaborate effectively with data analysts, and business stakeholders.
• Communication: Strong verbal and written communication skills to explain technical concepts to non-technical audiences.

1. Adaptability and Continuous Learning

Open to exploring new GCP features and rapidly adapting to changes in cloud technology

1. Core Data Engineering Skills

Big Data Technologies: Proficiency in using GCP's big data tools like

• BigQuery: For data warehousing and SQL analytics.
• Dataproc: For running Spark and Hadoop clusters.
• Airflow: For pipeline Orchestration
• Dataflow: For stream and batch data processing.(High level Idea)
• Pub/Sub: For real-time messaging and event ingestion.(High level Idea)
• Data Modeling: Experience/Knowledge on designing scalable, efficient data models for OLAP and OLTP use cases.

ETL/ELT Pipelines

Expertise in building automated, scalable, and reliable pipelines using custom Python/Scala solutions or Cloud Data Fusion.

1. Programming and Scripting

• Strong coding skills in Python, SQL , and optionally Java/Scala.
• Familiarity with APIs and SDKs for GCP services to build custom data solutions.

1. Cloud Infrastructure

• Understanding of GCP services such as Cloud Storage, Compute Engine, and Cloud Functions.
• Familiarity with Kubernetes (GKE) and containerisation for deploying data pipelines. (Optional but Good to have)

1. DevOps and CI/CD

• Experience setting up CI/CD pipelines using Cloud Build, GitHub Actions, or other tools.
• Monitoring and logging tools like Cloud Monitoring and Cloud Logging for production workflows.
• Graduate in Computer Science, or related field. 6+ years of experience in data engineering or related field.