Big Data Pipeline Architect

If you are passionate about building scalable and efficient data pipelines and enjoy working in a dynamic and challenging environment, we would love to hear from you

Position: Big Data Pipeline Architect

Location: New York, New York, United States

Company: Lobox

Job Type: Contract

Salary: $138 Per hour

About the Company: We are a global technology company that specializes in providing cutting-edge solutions for data management and analytics. Our mission is to help businesses harness the power of data to drive growth and innovation. We pride ourselves on our diverse and dynamic team, who are passionate about using technology to solve complex business challenges. We are currently seeking a Big Data Pipeline Architect to join our growing team.

Job Overview: As a Big Data Pipeline Architect, you will be responsible for designing, implementing, and maintaining our company���s data pipeline architecture. You will work closely with cross-functional teams to understand business requirements and design scalable and efficient solutions to process, store, and analyze large volumes of data. You will also be responsible for establishing and enforcing data governance policies and best practices.

Key Responsibilities:

• Design and implement end-to-end data pipeline architecture, including data ingestion, processing, storage, and analytics.
• Collaborate with business stakeholders to understand data requirements and translate them into technical specifications.
• Develop data models, data flow diagrams, and other documentation to communicate data pipeline design.
• Design and implement data quality checks and data validation processes to ensure the accuracy and completeness of data.
• Optimize data pipeline performance and scalability to meet growing business needs.
• Work with data engineers and data scientists to integrate data from various sources and build data pipelines for analytics and machine learning.
• Establish and maintain data governance policies and procedures to ensure data security, privacy, and compliance.
• Monitor data pipeline performance and troubleshoot issues to ensure smooth and uninterrupted data flow.
• Stay updated with the latest trends and advancements in big data technologies and recommend new tools and techniques to improve data pipeline efficiency.

Qualifications:

• Bachelor���s or Master���s degree in Computer Science, Information Systems, or a related field.
• Minimum of 5 years of experience in data engineering and architecture, with at least 2 years in a similar role.
• Strong experience with big data technologies such as Hadoop, Spark, Kafka, and NoSQL databases.
• Proficiency in programming languages such as Java, Python, or Scala.
• Experience with cloud-based data platforms such as AWS, Azure, or Google Cloud.
• Knowledge of data governance principles, data modeling, and data integration techniques.
• Excellent problem-solving and analytical skills.
• Strong communication and collaboration skills.
• Ability to work independently and in a team-oriented, fast-paced environment.

Join our team and be a part of our exciting journey towards transforming businesses through the power of data.