Spark Developer

If you are passionate about big data and have a strong background in Spark, we would love to hear from you

Job Title: Spark Developer

Location: New York, New York, United States

Salary: $100 Per hour

Overview: We are seeking a highly skilled and motivated Spark Developer to join our dynamic team. As a Spark Developer, you will be responsible for designing, developing, and maintaining complex data processing pipelines using Spark and other big data technologies. This role requires strong technical skills, a solid understanding of Spark architecture, and the ability to work in a fast-paced environment.

Key Responsibilities:

• Design and implement scalable and efficient data processing pipelines using Spark
• Collaborate with cross-functional teams to gather and analyze business requirements
• Develop and maintain ETL processes to ingest data from various sources into the data lake
• Optimize data models and data structures for improved performance
• Troubleshoot and debug issues with existing pipelines and data processes
• Ensure data quality and consistency by implementing data validation and quality checks
• Monitor and tune Spark jobs for optimal performance
• Stay updated with the latest advancements in Spark and other big data technologies
• Mentor and guide junior developers on best practices and coding standards
• Collaborate with data scientists to integrate machine learning models into data pipelines

Qualifications:

• Bachelor's or Master's degree in Computer Science, Engineering, or a related field
• Minimum of 3 years of experience in software development with at least 2 years of hands-on experience in Spark
• Strong knowledge of Spark architecture and concepts such as RDDs, DataFrames, and Datasets
• Experience with Hadoop ecosystem and related technologies such as Hive, HDFS, and YARN
• Proficient in programming languages such as Java, Scala, or Python
• Experience with SQL and database systems
• Familiarity with cloud platforms such as AWS, Azure, or Google Cloud
• Excellent problem-solving and analytical skills
• Ability to work independently and in a team environment
• Strong communication and interpersonal skills

Nice-to-Have:

• Experience with streaming technologies such as Kafka or Spark Streaming
• Knowledge of containerization technologies like Docker or Kubernetes
• Experience with Agile/Scrum methodologies
• Certifications in Spark or other big data technologies

We offer a competitive salary package, comprehensive benefits, and a collaborative work environment. As a Spark Developer, you will have the opportunity to work on cutting-edge technologies and make a significant impact on our data-driven organization.

Apply now and join our team of talented developers!