Job Title: Azure Databricks Engineer
Job Summary:
We are looking for an experienced Azure Databricks Engineer to join our team. The ideal candidate will have hands-on experience in designing, developing, and managing scalable data solutions using Azure Databricks. The role involves working collaboratively with cross-functional teams to build and maintain robust data pipelines and analytics solutions.
Key Responsibilities:
- Design and Implement Data Pipelines: Develop, deploy, and optimize data pipelines and ETL/ELT processes using Azure Databricks.
- Collaborate with Data Teams: Work closely with data analysts, data scientists, and data engineers to gather requirements and design solutions.
- Data Modeling and Management: Create and maintain data models that ensure data quality and governance.
- Optimize Performance: Tune Databricks clusters and notebooks to maximize efficiency and minimize cost.
- Integrate with Azure Ecosystem: Connect and integrate Databricks with other Azure services like Azure Data Lake Storage (ADLS), Azure Synapse Analytics, Azure SQL Database, and Azure Data Factory.
- Develop Scalable Solutions: Ensure solutions are scalable, maintainable, and aligned with best practices for data engineering.
- Support and Troubleshooting: Monitor jobs, handle exceptions, and resolve issues related to data pipelines and processing.
- Documentation: Create and maintain technical documentation for data workflows and solutions.
Technical Skills:
- Strong proficiency in Apache Spark and its applications within Azure Databricks.
- Experience with Python/Scala/Spark SQL for data processing.
- Expertise in integrating Databricks with Azure Data Lake Storage (ADLS), Azure Data Factory (ADF), and Azure Synapse.
- Knowledge of CI/CD pipelines using tools like Azure DevOps or GitHub Actions.
- Proficiency with data modeling concepts and SQL for building and querying data structures.
Additional Desirable Skills
- Certifications: Azure certifications such as Microsoft Certified: Azure Data Engineer Associate or Databricks Certification.
- Experience with machine learning workflows and frameworks within Databricks.
- Familiarity with Big Data technologies and distributed computing.
- Education: Bachelor's degree in computer science, Engineering, Data Science, or related field (or equivalent work experience).
Salary: 12,000 - 30,000 depending on experience
