User details
AJ
AJ
Alfred Jomma
@alfred_jomma_06jkv0636
Data Engineer
Anton, TX, United States
0Follower0Following0Mutual0Post
Member since: Jan 22, 2025• Data Engineer with over 7 years of experience designing, developing, and optimizing big data solutions, specializing in cloud technologies, data pipelines, and ETL processes.
• Proficient in building and maintaining large-scale data pipelines that handle massive datasets, ensuring efficient and scalable processing using industry-standard tools. . Extensive expertise with cloud platforms such as AWS, Azure, and Google Cloud Platform (GCP), with hands-on experience architecting cloud-based data solutions.
• Skilled in utilizing big data technologies like Hadoop, Apache Spark, Kafka, and Hive to process and manage large volumes of data in real-time and batch environments. Experienced in developing and automating ETL workflows using Python, SQL, and Scala, optimizing data flow across distributed systems.
• Expertise in cloud-based data warehousing solutions such as Snowflake, Redshift, and Azure Data Warehouse, with an emphasis on performance tuning and optimization.
• Proven ability to design scalable data architectures for data ingestion, transformation, and storage to support large-scale analytics and business intelligence systems. . Adept at implementing real-time data processing systems using Spark Streaming and Kafka, ensuring seamless data flow across applications.
• Strong experience in integrating diverse data sources, including APIs, flat files, relational databases, and external systems, to enable comprehensive data analysis.
• Highly skilled in data cleaning, validation, transformation, and standardization processes, ensuring high-quality data is available for analytics and reporting.
• Expertise in designing and implementing logical and physical data models that optimize storage, query performance, and resource management in data systems. . Hands-on experience with popular data orchestration and workflow management tools like Apache Airflow, Databricks, and Jupyter Notebooks for automating complex data workflows.
• Familiar with Agile development methodologies, using tools like JIRA, Confluence, and Scrum for collaborative project management and iterative development.
• Strong background in automating manual and repetitive tasks through shell scripting and Python, reducing operational costs and improving workflow efficiency.
• Deep understanding of data governance, privacy regulations, and best practices for ensuring data security and compliance within enterprise systems.
• Expertise in performance tuning for data processing platforms like Hive, Impala, and Spark, optimizing resource consumption and reducing processing time.
• Proven track record of migrating on-premise data
• Function as Data Engineer, assisted in leading the plan, building, and running states within the Enterprise Analytics Team.
• Lead architecture and design of data processing, warehousing and analytics initiatives.
• Engaged in solving and supporting real business issues with your Hadoop distributed File systems and Open-Source framework knowledge.
• Responsible for data governance rules and standards to maintain the consistency of the business element names in the different data layers.
• Involved in various phases of development analyzed and developed the system going through Agile Scrum methodology.
• Performed detailed analysis of business problems and technical environments and use this data in designing the solution and maintaining data architecture.
• Designed and implemented a recommendation system which leveraged Google Analytics data and the machine learning models and utilized Collaborative filtering techniques to recommend facilities for different patients.
• Built the data pipelines that will enable faster, better, data-informed decision-making within the business.
• Developed Spark scripts by using python and bash Shell commands as per the requirement.
• Worked on POC to check various cloud offerings including Google Cloud Platform (GCP).
• Developed a POC for project migration from on prem Hadoop MapR system to GCP. . Compared Self hosted Hadoop with respect to GCPs Data Proc, and explored Big Table (managed HBase) use cases, performance evolution.
• Used Terraform to reliably version and create infrastructure on GCP.
• Created resources, using GCP Terraform modules, and automated infrastructure management. . Build a program with Python and apache beam and execute it in cloud Data flow to run Data validation between raw source file and big query tables. . Used Rest API with Python to ingest Data from and some other site to Big Query. . Loaded and transformed large sets of structured, semi structured and unstructured data using Hadoop/Big Data concepts.
• Performed Data transformations in Hive and used partitions, buckets for performance improvements. . Optimized Hive queries to extract the customer information from HDFS.
• Involved in scheduling Oozie workflow engine to run multiple Hive jobs.
• Continuously monitor and manage data pipeline (CI/CD) performance alongside applications from a single console with GCP.
• Write a Python program to maintain raw file archival in GCS bucket.
• Designed Google Compute Cloud Data Flow jobs that move data within a 200 PB data lake.
• Implemented scripts that load Google Big Query data and run queries to export data.
• Implemented business logic by writing UDFs and configuring CRON Jobs.
• Providing 24/7 On-call Production Support for various applications. Environment: Hadoop 3.3, Spark 3.
Responsibilities
• Performed data engineering responsibilities using agile software engineering practices.
• Used Agile (SCRUM) methodologies for Software Development.
• Participated in Code Reviews, Enhancement discussion, maintenance of existing pipelines & systems, testing and bug-fix activities on-going basis.
• Worked closely with the business analysts to convert the Business Requirements into Technical Requirements and prepared low- and high-level documentation.
• Migrated data warehouses to Snowflake Data warehouse. . Configured Hadoop Ecosystems to read data transactions from HDFS and Hive. . Defined virtual warehouse sizing for Snowflake for different type of workloads. . Reviewing the explain plan for the SQLs in Redshift. . Imported data from RDBMS to HDFS and Hive using Sqoop on regular basis.
• Worked on Snowflake environment to remove redundancy and load real-time data from various data sources into HDFS using Kafka. . Imported data from AWS S3 into Spark RDD, Performed transformations and actions on RDD's.
• Written POCs in Python to analyze the data quickly before applying big data solutions to process at a scale.
• Responsible for data governance rules and standards to maintain the consistency of the business element names in the different data layers. . Migrated Matillion pipelines and Looker reports from Amazon Redshift to Snowflake data warehouse. . Utilized Matillion ETL solution to develop pipeline that extract and transform data from multiple sources and loading to Snowflake.
• Worked on Ingesting data by going through cleansing and transformations and leveraging AWS Lambda, AWS Glue and Step Functions.
• Integrated and automated data workloads to Snowflake Warehouse.
• Created tables in snowflake DB, loading and analyzing data using Spark-Scala scripts.
• Wrote complex Hive queries to extract data from heterogeneous sources (DataLake) and persist the data into HDFS.
• Developed ETL pipelines in and out of data warehouse using combination of Python and Snowflake's Snow SQL.
• Ingested data into HDFS using SQOOP and scheduled an incremental load to HDFS.
• Worked on Kafka REST API to collect and load the data on Hadoop file system and also used Sqoop to load the data from relational databases. . Extract Real time feed using Kafka and Spark Streaming and convert it to RDD and process data in the form of Data Frame and save the data as Parquet format in HDFS. . Used Spark-Streaming APIs to perform necessary transformations and actions on the data got from Kafka and Persists into HDFS.
• Created external tables with partitions using Hive, AWS Athena and Redshift.
• Developed the Pyspark code for AWS Glue jobs and for EMR.
• Created S3 buckets also managing policies for S3 buckets and Utilized S3 bucket and Glacier for storage and backup on AWS. . Develop Python and SQL used in the transformation process in Matillion. .
• As Data Engineer I have taken initiative and ownership to provide business solutions on time.
• Created High level technical design documents and Application design documents as per the requirements and delivered clear, well-communicated and complete design documents.
• Implemented Azure Data Factory (ADF) extensively for ingesting data from different source systems like relational and unstructured data to meet business functional requirements.
• Design and developed Batch processing and real-time processing solutions using ADF, Databricks clusters and stream Analytics.
• Created numerous pipelines in Azure using Azure Data Factory v2 to get the data from disparate source systems by using different Azure Activities like Move & Transform, Copy, filter, for each, Databricks etc.
• Maintain and provide support for optimal pipelines, data flows and complex data transformations and manipulations using ADF and PySpark with Databricks.
• Automated jobs using different triggers like Events, Schedules and Tumbling in ADF.
• Created, provisioned different Databricks clusters, notebooks, jobs and auto scaling.
• Created several Databricks Spark jobs with Pyspark to perform several tables to table operations.
• Performed data flow transformation using the data flow activity.
• Implemented Azure, self-hosted integration runtime in ADF.
• Developed spark programming code in Python Databricks workbooks.
• Improved performance by optimizing computing time to process the streaming data by optimizing the cluster run time.
• Developed Power BI reports using power query as a feed from SQL Server & different Data sources. . Ensure data accuracy, integrity, and reliability of both in back-end Power BI reports. . Created Linked services to connect the external resources to ADF.
• Wrote complex SQL queries including joins, correlated sub queries, scalar sub queries, views, Stored Procedures and Triggers
• Ensured the developed solutions are formally documented and signed off by business.
• Worked with team members to resolve any technical issue, Troubleshooting, Project Risk &Issue identification, and management.
• Worked on the cost estimation, billing, and implementation of services on the cloud.
• Worked as Data Analyst, analyzed data quality issues against source system and prepared a data quality document confirming all the source data quality.
• Worked with the application and Business Analyst team to develop requirements.
• Involved in Data profiling and performed Data Analysis based on the requirements, which helped in catching many Sourcing Issues upfront. . Used SQL tools to run SQL queries and validate the data loaded into the target tables.
• Created tables, views, sequences, indexes, constraints and generated SQL scripts for implementing physical data model.
• Used AWS services EC2 and S3 for small data sets processing and storage.
• Designed and developed architecture for data services ecosystem spanning Relational, NoSQL, and Big Data technologies.
• Involved in extensive data validation by writing several complex SQL queries.
• Involved in back-end testing and worked with data quality issues.
• Created dimensional model for reporting system by identifying required dimensions and facts using Erwin.
• Worked and extracted data from various database sources like XML and Flat files into the Data Stage. . Consumed XML messages using Kafka and processed the xml file using Spark streaming to capture UI updates.
• Evaluated data profiling, cleansing, integration and extraction tools.
• Involved in extensive Data validation by writing SQL queries and involved in back-end testing and worked with data quality issues.
• Responsible for the data architecture design delivery, data model development, review, approval and Data warehouse implementation.
• Involved in designing and developing Data Models and Data Marts that support the Business Intelligence Data Warehouse.
• Involved in extensive data validation by writing several complex SQL queries and involved in back-end testing and worked with data quality issues.
• Created SQL tables with referential integrity, constraints and developed queries using SQL, SQL*PLUS and PL/SQL.
• Performed GAP analysis of current state to desired state and document requirements to control the gaps identified.
• Managed all indexing, debugging and query optimization techniques for performance tuning using T-SQL.
• Generated comprehensive analytical reports by running SQL queries against current databases to conduct data analysis. . Wrote T-SQL statements for retrieval of data and involved in performance tuning of T- SQL queries and Stored Procedures.
• Developed pre-processing job using Spark Data frames to flatten Json documents to flat file.
• Developed Dashboard reports on Tableau.
Environment: SQL, PL/SQL, SAS, Microsoft Excel 2010, T-SQL, Pivot Tables, VLOOK up, triggers, Stored Procedures.