Data Engineer

Position Overview: 

We are seeking a highly skilled and motivated Data Engineer with a strong background in Machine Learning (ML) projects and extensive experience working with Databricks and Apache Spark. As a key member of our data team, you will play a crucial role in designing, implementing, and optimizing data pipelines, frameworks, and infrastructure to support ML initiatives. The ideal candidate should possess a deep understanding of data engineering concepts, excellent programming skills, and a passion for leveraging cutting-edge technologies to drive innovative ML solutions.

Responsibilities:

• Data Pipeline Development: Design, develop, and maintain robust, scalable, and efficient data
  pipelines using Databricks and Spark to extract, transform, and load both structured and
  unstructured data from various sources.
• ML Infrastructure: Collaborate with data scientists and ML engineers to build and maintain the
  infrastructure required for ML model training, evaluation, and deployment. This includes setting
  up distributed computing environments, managing clusters, and optimizing resource allocation.
• Data Quality and Integrity: Ensure data integrity and quality throughout the data lifecycle by
  implementing data validation checks, error handling mechanisms, and data monitoring
  processes.
• Performance Optimization: Identify performance bottlenecks in data pipelines and Spark jobs
  and work on optimizations to enhance processing speed and efficiency.
• Data Governance: Implement security protocols, access controls, and data governance policies
  to maintain data privacy and compliance with industry standards and regulations.
• Collaboration: Work closely with cross-functional teams, including data scientists, software
  engineers, and business analysts, to understand ML requirements, provide data engineering
  support, and deliver successful ML projects.
• Automation and Scalability: Develop automated solutions for repetitive tasks, and design data
  engineering systems that can scale seamlessly to handle large volumes of data.
• Troubleshooting and Support: Monitor data pipelines and Spark jobs, proactively identify
  issues, and provide timely support to maintain system availability and performance.
• Continuous Learning: Stay up-to-date with the latest advancements in data engineering, ML,
  and cloud technologies, and actively apply this knowledge to improve existing systems and
  processes.

Qualifications:

• Bachelor’s or higher degree in Computer Science, Engineering, or a related field.
• Minimum of 4 years of relevant experience as a Data Engineer working on ML projects with a
  strong focus on Databricks and Spark.
• Proven hands-on experience with Databricks and Apache Spark for data processing and ML
  workloads in a cloud environment.
• Solid understanding of data engineering principles, data modeling, ETL processes, and data
  warehousing concepts.
• Proficiency in programming languages like Python, Scala, or Java, and experience with SQL for
  data manipulation and querying.
• Familiarity with cloud platforms such as AWS, Azure, or Google Cloud Platform, and
  understanding of cloud-based data storage and computing services.
• Prior experience in building and managing ML infrastructure, including ML frameworks (e.g.,
  TensorFlow, PyTorch), model versioning, and deployment.
• Familiarity with other big data technologies like Hadoop, Hive, or Kafka is a plus.
• Strong analytical and problem-solving abilities with a keen eye for detail and a proactive
  approach to identifying and resolving issues.
• Excellent communication and teamwork skills to collaborate effectively with diverse teams and
  convey complex technical concepts to non-technical stakeholders.