User details
TW
TW
Tommy Wang
@tommy_wang_v6k9z7g31
Data Engineer
San Antonio, TX, United States
0Follower0Following0Mutual0Post
Member since: Jan 29, 2025• Around 11 years of IT experience as a Data Engineer in a variety of industries, which includes hands on experience in Hadoop, MapReduce, Hive, Spark (Pyspark), Azure, AWS, Sqoop, Snowflake, Teradata, Oracle, RDMS, Python, Scala, ETL and Data Visualization.
• Experienced in Databricks & Big Data Technologies mainly on Data and Delta Lake Implementation.
• Excellent understanding of all stages in a typical SDLC like Requirement Analysis, Design, Programming, Project Status Review (PSR), Unit Testing, Integration Testing, Support
• Expert in designing and implementing efficient data processing pipelines using Spark, Hadoop, and cloud based ETL tools such as AWS Glue and Azure Data Factory. . Extensive knowledge in Hadoop, Spark, Pyspark, Hive, Impala, Sqoop, RDBMS (Oracle, Maria DB).
• Experienced in Spark with Hadoop platform for processing billions of data which uses in-memory data processing. . Experience in Data governance & Data modelling, use ER studio analyse the relationship database.
• Worked with NoSQL databases like HBase in creating tables to load large sets of semi structured data.
• Worked with Python and R Libraries like NumPy, Pandas, Scikit-Learn, SciPy, and TensorFlow.
• Proficient in managing entire data science(Al) project life cycle and actively involved in all the phases of project life cycle including data acquisition, data cleaning, data engineering, features scaling, features engineering, statistical modelling (decision trees, regression models, neural networks, SVM, clustering), dimensionality reduction using Principal Component Analysis and Factor Analysis, testing and validation using ROC plot, K- fold cross validation and data visualization.
• Good knowledge of Data Visualization including producing tables, graphs, and listings using various procedures and tools such as Tableau.
• Working knowledge of Integration Services (SSIS), Reporting Services (SSRS), and Analysis Services (SSAS).
• Worked in extracting data from multiple sources to Power BIData Preparation.
• Managed and optimized cloud-based infrastructure on AWS Azure GCP to ensure high availability and scalability of data processing and machine learning workloads. . Supported migration of data of loading modes transaction, master from RDMS to
• Worked for VF Corporation loyalty team, design, build, and maintain data infrastructure and models that support VF Corporation's different loyalty programs, integrate data from various sources and monitor the performance of loyalty program initiatives using key performance indicators, including enrolment, order and points redemption management, Management of loyalty data up and downstream, with understanding of reporting requirements for loyalty as a critical business driver, such as OMS, CDP, GDF, Driving transformation of loyalty program enablement through improved technology, processes, data management, and use case refinement for VF brands Vans, The North Face and Timberland. . Write Scope script to extract the data of user subscription and usage from Cosmos and Analysis different features. Created the connection between Cosmos and Azure data lake storage gen1 and transferred the data-to-Data Lake. Used DataBricks, python to process data, Analysis the relationship and relevance about the features and subscription. Used Logistic Regression, Random Forest to build the Machine learning model to identify and target 'at risk' users. Deploy and Operate the Machine
learning model on Azure (DevOps), help the Microsoft Xbox marketing team to optimize and personalize campaigns to move members from high churn risk to low-risk segment.
• Worked for the Walmart supply chain e-commerce team. Write shell script to extract data from Kafka and transfer to GCP buckets with data mapping and transformation, create different schemas in Big Query and use ETL tool (Automic) to write pipeline and land data. Write query to get data and create dashboard and do data quantitative analysis.
• Assist in building Data pipelines to improve data quality and facilitate iterations for accommodating new user requirements.
• Proactively working on building data lineage which would display incoming and outgoing reference of an asset by extensive use of data pipelines and other Meta tools such as Uni dash, Dai query, Diffs and Sev.
• Performed data transformation and processing tasks within Snowflake by leveraging Snowflake's built-in SQL capabilities. Also, data quality and consistency by implementing data validation checks, data cleansing, and normalization processes.
• Design the data bricks notebooks using Pyspark for data extraction, transformation and landing the data into the data bricks Delta Table.
• Utilize Azure DevOps for continuous integration and deployment (CI/CD) pipelines, integrating Kubernetes deployments and configuration management for automated deployment, monitoring, and management of data solutions. . Using Pandas, NumPy, SciPy, and Scikit-learn in Python for scientific computing and data analysis.
• Working extensively with Databricks to build scalable and efficient data processing and transform data pipelines to handle high-volume data ingestion, transformation, and aggregation. .
Chartered and headquartered in Omaha, Nebraska, United states, First National provides corporate banking, investment banking, retail banking, wealth management and consumer lending services at locations in Nebraska, lowa, colorado, Texas, Kansas, South Dakota and Illinois.
. Helping executives to understand the usage of Azure services which mainly focus on Machine Learning Model where Design Develop and Deploy all the Azure Machine Learning Model with their activities and their ETL data pipeline in Azure Data Factory which helps to analyse the data to track the usage of Azure services.
• Developed and optimized data warehouse for datasets using Azure Data Lake Storage Explorer and Kusto Data Explorer
• Support senior management by providing automatic metrics reporting and ad-hoc deep dive analysis, managed SLA, implemented & automated reported intelligent analytics infrastructure of ML Models using SQL Server, Excel, and Power BI. . Used Hive & Spark for data processing & Flume to transfer log files from multiple sources to HDFS.
• Developed Stream Processing data pipelines using Spark Streaming, and Kafka. . Extract Transform and Load data from Sources Systems to Azure Data Storage services using a combination of Azure Data Factory, T-SQL, Spark SQL, and U-SQL Azure Data Lake Analytics. . Data Ingestion to one or more Azure Services - (Azure Data Lake, Azure Storage, Azure SQL, Azure DW) and processing the data in In Azure Databricks.
• Implemented real time data ingestion pipeline to ingest data from trip booking system into Hadoop and HBase for analytical purposes using Kafka, Python, Pyspark, HBase enabling to load 100 million records per day into Data Lake.
• Worked in close partnership with cross-functional teams across organization to define business rules and cleansing requirements for data transformation process. . Responsible for technical coordination including writing SQL query and C# Script logic and leveraging assets to build and deploy email campaigns.
• Responsible for DevOps activities such as code maintenance in GIT, Jenkins.
• Generated discover reports using Power BI and presented campaign engagement statistics to business owners to direct future strategies and initiatives.
Environment: Azure Data Factory, Kafka, Pyspark, SQL Server Management Studio, Visual Studio, Power BI
• Created action filters, calculations, parameters, and calculated sets for preparing dashboards and worksheets.
• Designed and developed Power BI graphical and visualization solutions with business requirement documents and plans for creating interactive dashboards and developed various DAX measures in Power BI to satisfy business needs.
• Transformed stored data by writing Spark jobs based on business requirements.
• Developed SSRS reports, SSIS packages, and data pipelines using Kafka for ETL jobs to optimize the Tensor Flow model, and implemented efficient Hive SQL, and Spark SQL.
• Collaborated with data engineers and operation team to implement ETL process, wrote and optimized SQL queries to perform data extraction to fit analytical requirements.
• Performed preliminary data analysis using descriptive statistics and handled anomalies such as removing duplicates and imputing missing values.
• Performed data mining on large datasets (mostly structured) and raw text data using different Data Exploration techniques.
• Developed Spark applications using Pyspark and Spark-SQL for data extraction, transformation, and aggregation from multiple file formats for analysing & transforming the data to uncover insights into the customer usage patterns.
• Build and maintain Snowflake databases, including data modelling, partitioning, and clustering for optimized performance. . Wrote ETL jobs to read from web APIs using REST and HTTP calls and loaded into HDFS using Python. . Built Data Integration and Workflow Solutions for data warehousing using SQL Server Integration Service and OLAP.
Environment: Python, Azure PaaS service, PowerBI, SSRS, T-SQL, Spark SQL, U-SQL, SOAP APIs, REST APIs, Pyspark, Spark-SQL, JSON, Azure Data Factory, Azure Data bricks