Key Responsibilities:

Design, develop, and maintain Spark applications for data processing and analysis.
Collaborate with data engineering and data science teams to understand business requirements and design efficient data pipelines.
Optimize Spark jobs for performance, scalability, and reliability.
Troubleshoot and resolve issues related to Spark application failures.
Develop and maintain ETL processes to ingest and transform data from various data sources.
Implement data governance and security measures to ensure data integrity and compliance.
Develop and maintain technical documentation for Spark applications.
Mentor and train junior team members on Spark development best practices.
Keep up-to-date with the latest advancements in Spark and big data technologies and identify opportunities for improvement.

Qualifications:

Bachelor's or Master's degree in Computer Science, Engineering, or related field.
Minimum of 5 years of experience in developing Spark applications.
Extensive hands-on experience with Apache Spark, Scala, and Python.
Solid understanding of big data concepts and distributed computing.
Experience with cloud-based data platforms such as AWS or Azure.
Experience with SQL and NoSQL databases.
Strong knowledge of software engineering principles and design patterns.
Excellent problem-solving and analytical skills.
Strong communication and teamwork abilities.
Ability to work in a fast-paced and dynamic environment.

Hashtags