User details
NK
NK
Naveen Kannedari
@naveen_kannedari_xje1oyq10
Azure Engineer
San Antonio, TX, United States
1Follower0Following0Mutual0Post
Member since: Jan 21, 2025• Data Engineer with 4+ years of experience in building on-premises and cloud-based data solutions using cutting edge tools like Azure Data Factory, Data bricks, Spark, SQL, and Python
• Adept at using Databricks for developing ETL and ELT processes using PySpark, Spark-SQL, and Scala
• Proficient in building and orchestrating ETL pipelines using Azure Data Factory and SQL Server Integration Services (SSIS)
• Experienced in implementation of Lakehouse architecture on Azure using Azure Data Lake, Delta Lake, Delta Tables, and Databricks
• In-depth understanding and usage of various Databricks optimization techniques like Partitioning, Bucketing, Adaptive Query Execution, DAG, Dynamic File Pruning, Data Skipping, Optimize, Bin-packing, Z-Order clustering on delta tables.
• Well versed in data warehousing and normalization techniques for architecting data models including star schema, +snow-flake schema, and slowly changing dimensions etc.
• Experience in provisioning access to Azure Data Lake from Databricks using Secret Scope and mounting the storage account into Databricks.
• Experienced in large data migration projects from on-prem databases to Azure SQL and Azure Data warehouse.
• Experienced in designing and building data models for OLTP and OLAP applications for a variety of business domains
• Expert in writing efficient SQL queries and optimizing poor performing queries and stored procedures.
• Solid experience in working as a data analyst to manage, document, analyze, and resolve data related issues. . Pragmatic and proactive in performance tuning on OLTP and Data Warehouse systems (OLAP) for fast and efficient data load and data retrieval
• Develop, design data models, data structures and ETL jobs for data acquisition and manipulation purposes.
• Develop deep understanding of the data sources, implement data standards, maintain data quality, and master data management.
• Worked with various file formats like CSV, JSON, Parquet, and Snappy Parquet.
• Used Python and Pyspark in Synapse notebooks to efficiently process huge amounts of data and load them into delta lake silver layer
• Utilized Delta caching for faster data processing using Databricks and PySpark.
• Worked on improving poorly performing PySpark code by using coalesce, repartition, query hints, broadcast joins, and delta caching.
• Involved in optimized Delta tables by using Optimize command and applying Z-Order Clustering. . Transformed batch data from several tables containing tens of thousands of records from SQL Server, MySQL, PostgreSQL, and csv file datasets into data frames using PySpark
• Performed ETL operations in Azure Databricks by connecting to different relational database source systems using JDBC connectors.
• Developed Python scripts to do file validations in Databricks and automated the process using ADF.
• Developed an automated process in Azure cloud which can ingest data daily from web service and load into Azure Data Lake Gen2.
• Used Logic App to take decisional actions based on the workflow.
• Developed custom alerts using Azure Data Factory, SQLDB and Logic App.
• Developed Databricks ETL pipelines using notebooks, Spark Data frames, SPARK SQL, and python scripting.
• Developed complex SQL queries using stored procedures, common table expressions (CTEs), temporary table to support Power BI reports.
• Worked with enterprise Data Modeling team on creation of Logical models.
• Development level experience in Microsoft Azure providing data movement and scheduling functionality to cloud- based technologies such as Azure Blob Storage and Azure SQL Database.
• Worked on developing JSON Scripts for deploying the Pipeline in Azure Data Factory (ADF) that process the data.
• Independently manage development of ETL processes - development to delivery.
• Develop proof of concepts (POC) using Azure Cloud services such as Azure Data factory (ADF), Logic Apps, Data Bricks, Azure SQL DB, Power BI and Azure Data Lake storage (ADLS) to validate proposed solutions and seek feedback from stakeholders.
• Working on getting the data from on prem SQL server database to a Raw Zone in Azure Data Lake by using Azure Data Factory.
• Used Parquet file formats on Azure Blobs to store raw data.
• Monitored the SQL scripts and modified them for improved performance using PySpark SQL
• Develop Azure Databricks notebooks to apply business transformations and perform data cleansing operations. Used spark, python, delta while reading data from parquet files in ADLS to Data frame and writing to destination ADLS location. Partitioned data and implemented schema drift by using metadata JSON file before writing data to destination location in ADLS.
• Implemented ETL platform using Azure Data Factory, Data Bricks, Data Lake, Azure SQL.
• Create External tables in Azure SQL Database for data visualization and reporting purpose.
• Performance improvement of existing views and procs and identify deadlocks, missing indexes and alter indexes where necessary
• Created T-SQL Stored Procedures, User Defined Functions, Indexes, Views, Constraints, Triggers and wrote complex queries using Joins, Aggregate Functions, Sub-queries, Derived tables and CTEs
• Implemented SCD type1, SCDType2 logic using Azure data bricks for Delta tables.
• Extract Transform and Load data from Sources Systems to Azure Data Storage services using a combination of Azure Data Factory, T-SQL, Azure Databricks, and Azure Data Lake Analytics.
• Created Pipelines in ADF using Linked Services/Datasets/Pipeline to Extract, Transform and load data from different sources like Azure SQL, Blob Storage, and Azure SQL Data warehouse.
• Working on data Ingestion & transformation from different source systems using Blob Storage/ADLS into Data bricks and processing.
• Develop and maintain application code in Databricks by using appropriate data structures and algorithms for optimal performance and better storage that increase the speed and consistency of the application.
• Created CI/CD Pipelines in azure Devops to build and deploy data factory ARM templates . Loaded the data into facts and dimensions based on Data modeling using Azure Data factory
• Scheduling ADF Pipelines as per the data dependency
• Used Key Vault to maintain secrets in Azure environment.
• Used Azure Devops to maintain Product Backlog Items, User Stories, Tasks and Bugs.
• Merging code to GIT to Azure Devops to maintain on distributed server.