| | | 20,000+ Fresh Resumes Monthly | |
|
|
| |
Click here or scroll down to respond to this candidatePhone: PHONE NUMBER AVAILABLE Email: EMAIL AVAILABLEPROFESSIONAL SUMMARY10+ Years of experience in Data Engineering, with expertise in designing and implementing scalable data ingestion pipelines using Azure Data Factory, including advanced features like copy activity, lookup activity, and integrating ADLS Gen2 for efficient data storage and management.Expertise in integrating on premise and cloud-based data sources using Azure Data Factory, applying transformations, and loading data into Snowflake, while making use of ADLS Gen2 for largescale data storage and copy activity for efficient data movement.Experienced data professional with a strong background in management of ETL data pipelines, ensuring scalability and smooth operations, utilizing compression techniques to optimize storage and transfer speeds.Proficient in optimizing query techniques and indexing strategies to enhance data fetching efficiency, including experience with Vertica and Teradata for high performance data analytics.Skilled in utilizing SQL queries, including DDL, DML, and various database objects, for data manipulation and retrieval, leveraging linked services in Azure Data Factory to connect disparate data sources seamlessly.Strong knowledge of data warehousing techniques, including data cleansing, Slowly Changing Dimension handling, surrogate key assignment, and change data capture for Snowflake modeling, utilizing lookup activity to maintain data accuracy and integrity.Experienced in designing and implementing scalable data ingestion pipelines using tools such as Apache Kafka, Apache Flume, and Apache Nifi, with an emphasis on data processing through EventHub.Proficient in developing and maintaining ETL/ELT workflows using technologies like Apache Spark, Apache Beam, or Apache Airflow for efficient data extraction, transformation, and loading processes, incorporating Event Queues for event driven data processing.Skilled in implementing data quality checks and cleansing techniques to ensure data accuracy and integrity throughout the pipeline, applying compression techniques to optimize data storage and processing.Experienced in building and optimizing data models and schemas using technologies like Apache Hive, Apache HBase, or Snowflake for efficient data storage and retrieval for analytics and reporting, integrating ADLS Gen2 for enhanced data management.Strong proficiency in developing ELT/ETL pipelines using Python and Snowflake Snow SQL, with extensive use of copy activity and lookup activity in Azure Data Factory for robust data integration and transformation processes.Collaborative team member, working closely with Azure Logic Apps administrators and DevOps engineers to monitor and resolve issues related to process automation and data processing pipelines, leveraging linked services for seamless integration.Configured the ADF jobs, Snow SQL jobs triggering in Matillion using python.Scheduled the loading of all staging, intermediate, and final core tables into Snowflake using the Matillion tool.Experienced in optimizing code for Azure Functions to extract, transform, and load data from diverse sources, incorporating EventHub for data streaming capabilities.Strong experience in designing, building, and maintaining data integration programs within Hadoop and RDBMS environments, utilizing various file formats like Avro, Parquet, Sequence, Json, ORC, and text for loading data, parsing, gathering, and performing transformations with ADLS Gen2.Good experience in Hortonworks and Cloudera for Apache Hadoop distributions, leveraging Event Queues for efficient data processing and management.Designed and created Hive external tables using shared meta store with Static & Dynamic partitioning, bucketing, and indexing, enhancing query performance in Vertica and Teradata environments.Exploring with Spark improving the performance and optimization of existing algorithms in Hadoop using Spark context, Spark SQL, Data Frame, pair RDDs, and incorporating compression techniques for data efficiency.Proficient in implementing CI/CD frameworks for data pipelines using tools like Jenkins, ensuring efficient automation and deployment, with linked services facilitating continuous integration and delivery in cloud and hybrid environments.TECHNICAL SKILLSAzure ServicesAzure data Factory, Airflow, Azure Data Bricks, Azure Logic Apps, Functional App, Snowflake, Azure DevOpsBig Data TechnologiesMapReduce, Hive, PySpark, Scala, Kafka, Spark streaming, Oozie, Sqoop, ZookeeperHadoop DistributionCloudera, Horton WorksLanguagesSQL, PL/SQL, Python, HiveQL, Scala.Operating SystemsWindows (XP/7/8/10), UNIX, LINUX, UBUNTU, CENTOS.Build Automation toolsAnt, MavenVersion ControlGIT, GitHub.IDE & Build Tools, DesignEclipse, Visual Studio.DatabasesMS SQL Server 2016/2014/2012, Azure SQL DB, Azure Synapse, MS Excel, MS Access, Oracle 11g/12c, Cosmos DBWORK EXPERIENCERole: Azure Snowflake Data Engineer Oct 2022 to till now.Client: Charter Communications, St. Louis, Missouri.Responsibilities:Developed a Promotion Engine application, focusing on optimizing customer engagement through eligibility assessment, offer validation, and discount application using Azure's cloud services.Designed and implemented data ingestion pipelines using Azure Data Factory, facilitating the seamless integration of diverse data sources such as SQL databases, Oracle database, CSV files, and REST APIs into Azure Blob Storage.Developed data processing workflows using Azure Databricks, leveraging Spark for distributed data processing and transformation tasks.Implemented data quality checks and data cleansing techniques to ensure the accuracy and integrity of the data throughout the pipeline, using Azure Data Factory and Databricks.Developed end-to-end ETL data pipelines, ensuring scalability and smooth functioning. This included extensive use of copy activity for data movement and lookup activity for data validation, leveraging linked services to connect on-premises and cloud data sources.Implemented optimized query techniques and indexing strategies to enhance data fetching efficiency, utilizing SQL queries, and incorporating ADLS Gen2 for scalable storage solutions.Integrated Snowflake with Azure cloud services, establishing a secure and efficient data warehousing solution that enabled stakeholders to generate insightful reports for strategic marketing analysis.Designed and implemented real-time data processing solutions using Kafka and Spark Streaming, enabling the ingestion, transformation, and analysis of high-volume streaming data.Integrated PySpark with Azure Data Factory and Azure Blob Storage to seamlessly ingest and process data, enabling seamless data integration and interoperability within the Azure ecosystem.Developed Spark core and Spark SQL scripts using Scala for faster data processing.Developed data warehousing techniques, data cleansing, Slowly Changing Dimension (SCD) handling, surrogate key assignment, and change data capture for Snowflake modelling.Optimized PySpark jobs for performance by leveraging techniques such as partitioning, caching, and broadcast joins, reducing processing times and resource utilization, thereby improving overall system efficiency.Conducted performance tuning and capacity planning exercises to ensure the scalability and efficiency of the data infrastructure.Optimized code for Azure Functions to extract, transform, and load data from diverse sources, including databases, APIs, and file systems, making strategic use of ADLS Gen2 for efficient data storage and management.Developed complex SQL queries and data models in Azure Synapse Analytics to integrate big data processing and analytics capabilities, enabling seamless data exploration and insights generation.Built and optimized data models and schemas using technologies like Apache Hive, Apache HBase, or Snowflake to support efficient data storage and retrieval for analytics and reporting purposes, with copy activity streamlining data movements.Created ETL transformations and validations using Spark SQL/Spark Data Frames with Azure Databricks and Azure Data Factory, employing lookup activity to ensure data accuracy and consistency.Integrated GitHub repositories with Azure services, such as Azure DevOps or Azure Pipelines, to enhance collaboration and automate deployment workflows within the Azure ecosystem.Designed and deployed interactive Power BI dashboards that provided real-time insights into customer eligibility, promotional effectiveness, and discount utilization, enhancing decision-making processes for marketing and sales teams.Collaborated with Azure DevOps team into the project workflow, improving code quality and project management efficiency through continuous integration/continuous deployment (CI/CD) pipelines.Actively collaborated with data analysts and business stakeholders to understand reporting requirements, facilitating the development of customized reports that drove strategic business decisions.Environment: Azure Databricks, Data Factory, Logic Apps, Functional App, Snowflake, MS SQL, Oracle, Spark, Hive, SQL, Python, Scala, PySpark, Shell scripting, GIT, JIRA, Jenkins, Kafka, ADF Pipeline, Power Bi.Role: Azure Snowflake Data Engineer Jan 2021 to Sep 2022Client: Wells Fargo.Responsibilities:Designed data engineering initiatives within an Azure Kubernetes Service (AKS) environment at Wells Fargo, with a focus on ensuring reliable, scalable, and efficient data operations.Designed and implemented end-to-end data pipelines, seamlessly integrating Azure services such as SQL Database, Data Lake Storage, and Data Factory.Implemented efficient data integration solutions to seamlessly ingest and integrate data from diverse sources, including databases, APIs, file systems, and Teradata, using tools like Apache Kafka, Apache NiFi, Azure Data Factory, and leveraging Event Hubs for data streaming.Data Ingestion to one or more Azure Services (Azure Data Lake, Azure Storage, Azure SQL, Azure DW) and processing the data in Azure Databricks, incorporating data from Event Hubs to facilitate real-time analytics and insights.Enhanced Spark performance by optimizing data processing algorithms, leveraging techniques such as partitioning, caching, broadcast variables, and compression techniques to reduce data size and improve execution speed.Worked on Microsoft Azure services like HDInsight Clusters, BLOB, Data Factory, Logic Apps, and utilized Event Hubs for collecting large streams of data from various sources, enhancing the capability for data processing.Perform ETL using Azure Data Bricks, migrated on-premises Oracle ETL processes to Azure Synapse Analytics, applying compression techniques to optimize storage and querying speeds.Leveraged Snowflake's native support for SQL, Python, and other languages to perform advanced analytics and data processing tasks within AKS.Worked on migrating SQL database to Azure data lake, Azure data lake analytics, Azure SQL Database, Data Bricks, Azure SQL Data warehouse, and integrated Teradata databases to ensure seamless data synchronization and reporting.Controlling and granting database access and migrating on-premises databases to Azure data lake store using Azure Data Factory, including databases from Teradata using efficient data transfer methods.Data transfer using Azure Synapse and Polybase, with a focus on implementing compression techniques to enhance data transfer efficiency and reduce storage costs.Leveraged Snowflake's automatic scaling capabilities to adapt to changing demands, ensuring optimal performance and resource utilization.Deployed and optimized Python web applications to Azure DevOps CI/CD, focusing on development while integrating data from Event Queues to manage application events and triggers effectively.Developed enterprise level solutions using batch processing and streaming frameworks, using Spark Streaming, Apache Kafka, and incorporating Event Queues to efficiently manage event driven data workflows.Designed and implemented robust data models and schemas to support efficient data storage, retrieval, and analysis using technologies like Apache Hive, Apache Parquet, Snowflake, and applied compression techniques for optimized storage.Developed and maintained end-to-end data pipelines using Apache Spark, Apache Airflow, or Azure Data Factory, ensuring reliable and timely data processing and delivery, including integration with Teradata for comprehensive data analysis.Collaborated with cross functional teams to gather requirements, design data integration workflows, and implement scalable data solutions, leveraging Event Hubs to capture and process event streams in real time.Provided production support and troubleshooting for data pipelines, identifying, and resolving performance bottlenecks, data quality issues, and system failures, with a focus on optimizing data flows from Event Queues.Environment: Azure Databricks, Data Factory, Logic Apps, Functional App, Snowflake, MS SQL, Oracle, HDFS, MapReduce, YARN, Spark, Hive, SQL, Python, Scala, PySpark, Spark Performance, data integration, data modeling, data pipelines, production support, Shell scripting, GIT, JIRA, Jenkins, Kafka, ADF Pipeline, Power Bi.Role: Data Engineer Oct 2018 to Dec 2020Client: UBS, Weehawken, NJ.Responsibilities:Designed and setup Enterprise Data Lake to provide support for various uses cases including Analytics, processing, storing, and reporting of voluminous, rapidly changing data.Responsible for maintaining quality reference data in source by performing operations such as cleaning, transformation and ensuring Integrity in a relational environment by collaborating closely with the stakeholders & solution architect.Worked on creating tabular models on Azure analytic services for meeting business reporting requirements.Data Ingestion to one or more cloud Azure Services (Azure Data Lake, Azure Storage, Azure SQL, Azure DW) and cloud migration processing the data in Azure Databricks.Creating pipelines, data flows and complex data transformations and manipulations using ADF and PySpark with Azure Databricks.Working with Azure BLOB and Data Lake storage and loading data into Azure SQL Synapse analytics (DW).Developed Python, PySpark, Bash scripts logs to Transform, and Load data across on premises and cloud platform.Worked on Apache Spark Utilizing the Spark, SQL, and Streaming components to support the intradayand real-time data processing.Set up and developed Kerberos authentication principles to establish secure network communication on cluster and testing of HDFS, Hive, Pig and Map Reduce to access cluster for new users.Used Spark SQL for Scala & amp, Python interface that automatically converts RDD case classes to schema RDD.Import the data from diverse sources like HDFS/HBase into Spark RDD and perform computations using PySpark to generate the output response.Implementing different performance optimization techniques such as using distributed cache for small datasets, partitioning, and bucketing in hive, doing map side joins etc.Good knowledge on Spark platform parameters like memory, cores, and executorsDeveloped reusable framework to be leveraged for future migrations that automates ETL from RDBMS systems to the Data Lake utilizing Spark Data Sources and Hive data objects.Importing & exporting databases using SQL Server Integrations Services (SSIS) and Data Transformation Services (DTS Packages).Environment: Azure, Azure Data Factory, Databricks, PySpark, Python, Apache Spark, HBase, HIVE, SQOOP, Snowflake, SSRS, Tableau.Role: Big data Developer Dec 2017 to Sep 2018Client: Mayo Clinic, Rochester, MNResponsibilities:Designed and developed applications on the data lake to transform the data according business users to perform analytics.In depth understanding/ knowledge of Hadoop architecture and various components such as HDFS, application manager, node master, resource manager name node, data node and map reduce concepts.Involved in developing a Map Reduce framework that filters bad and unnecessary records.Involved heavily in setting up the CI/CD pipeline using Jenkins, Maven, Nexus, GitHub.Developed data pipeline using flume, SQOOP, pig and map reduce to ingest customer behavioral data and purchase histories into HDFS for analysis.Used Spark SQL to load JSON data and create schema RDD and loaded it into Hive tables handled structured data using Spark SQLUsed HIVE to do transformations, event joins and some preaggregations before storing the data onto HDFS.The Hive tables created as per requirement were internal or external tables defined with appropriate static and dynamic partitions, intended for efficiency.Implemented the workflows using Apache OOZIE framework to automate tasks.Developing design documents considering all approaches and identifying the best of them.Written Map Reduce code that will take input as log files and parse them and structure them in tabular format to facilitate effective querying on the log data.Developed scripts and automated data management from end to end and sync up b/w all the Clusters.Implemented Fair schedulers on the Job Tracker to share the resources of the cluster for the Map Reduce jobs given by the users.Environment: Cloudera CDH three-fourths, Hadoop, HDFS, MapReduce, Hive, Oozie, Pig, Shell Scripting, MySQL.Role: Data Warehouse & ETL Developer June 2012 to July 2016Client: Semantic Space, Hyderabad, India.Responsibilities:Developing complex store procedures, efficient triggers, required functions, creating indexes and indexed views for performance.Monitoring SQL Server Performance tuning in SQL ServerDesigning ETL data flows using SSIS; creating mappings/workflows to extract data from SQL Server and Data Migration and Transformation from Access/Excel Sheets using SQL Server SSIS.Worked on Building Cubes and Dimensions with different Architectures and Data Sources for Business Intelligence and writing MDX Scripting.Used Data warehouse for developing Data Mart which for feeding downstream reports, development of User Access Tool using which users can create ad hoc reports and run queries to analyze data in the proposed Cube.Deployed the SSIS Packages and created jobs for efficient running of the packages.Expertise in creating ETL packages using SSIS to extract data from heterogeneous database and then transform and load into the data mart.Involved in creating SSIS jobs to automate the reports generation, cube refresh packages.Great Expertise in Deploying SSIS Package to Production and used different types of Package configurations to export various package properties to make package environment independent.Experienced with SQL Server Reporting Services (SSRS) to author, manage, and deliver both paper based and interactive WEB based reports.Developing on Dimensional Data Modeling for Data Mart design, identifying Facts and Dimensions, and fact tables, dimension tables, using Slowly Changing Dimensions (SCD).Thorough knowledge of Features, Structure, Attributes, Hierarchies, Star Schemas of Data Marts.Developed SSAS Cubes, Aggregation, KPIs, Measures, Partitioning Cube, Data Mining Models, Deploying, and Processing SSAS objects.Creating Ad hoc reports and reports with complex formulas and querying the database for Business Intelligence.Flexible, enthusiastic, and project-oriented collaborator with excellent written, verbal communication and leadership skills to develop creative solutions for challenging client needs.Environment: MS SQL Server 2016, Visual Studio 2017/2019, SSIS, Share point, MS Access, Team Foundation server, Git.EDUCATIONCompleted Bachelors in Osmania University, Hyderabad, India.PRANAVI KUNTAAzure Snowflake Data Engineer |
