We are seeking a skilled and innovative Big Data Engineer to join our dynamic team. The Big Data Engineer will be responsible for designing, developing, and maintaining our big data infrastructure, ensuring the efficient processing and analysis of large datasets. The ideal candidate should have a strong background in big data technologies, and data engineering, and a passion for leveraging data to drive business insights. If you are a creative problem solver with expertise in big data technologies, we invite you to apply for this exciting opportunity.
Key Responsibilities:
- Design and implement scalable and efficient big data architectures.
- Evaluate and select appropriate big data technologies based on project requirements.
- Develop data pipelines for the ingestion and processing of large datasets.
- Ensure data quality and reliability throughout the processing flow.
- Implement and optimize data storage solutions for structured and unstructured data.
- Manage and maintain big data storage infrastructure.
- Perform data transformation and manipulation using big data processing frameworks.
- Optimize data transformation processes for performance and efficiency.
- Identify and resolve performance bottlenecks in big data systems.
- Implement optimizations for data processing speed and resource utilization.
- Collaborate with data scientists, analysts, and other stakeholders to understand data requirements.
- Work closely with IT teams to integrate big data solutions into the overall technology ecosystem.
- Implement security measures to protect big data systems and sensitive information.
- Ensure compliance with data privacy and regulatory requirements.
Qualifications:
- Minimum of 5 years of experience in big data engineering or a related field.
- Bachelor's or Master's degree in Computer Science, Information Technology, or a relevant field.
Skills and Competencies:
- Proficiency in big data technologies such as Apache Hadoop, Spark, and Kafka.
- Experience with distributed computing and storage systems.
- Strong programming skills in languages like Java, Python, or Scala.
- Problem-solving and analytical thinking.
