| | | 20,000+ Fresh Resumes Monthly | |
|
|
| |
Click here or scroll down to respond to this candidateCandidate's Name
Data EngineerEMAIL: EMAIL AVAILABLE CONTACT: PHONE NUMBER AVAILABLEPROFESSIONAL SUMMARY:Over 5+ years of experience in Information Technology as a Data Engineer, with hands-on expertise in Azure, Big Data technologies, and advanced SQL.Designed and implemented robust ETL pipelines to facilitate efficient data extraction, transformation, and loading in Snowflake.Proficient in the Hadoop ecosystem including Spark, Kafka, HBase, MapReduce, Python, Scala, Pig, Impala, Sqoop, Oozie, Flume, and Storm. Experienced in working with Spark SQL, Spark Streaming, and leveraging Core Spark API to build robust data pipelines.Experience in building big data solutions using Lambda Architecture with Cloudera distribution of Hadoop, integrating with Azure services.Strong knowledge and experience in implementing Big Data solutions in Azure Databricks for processing and managing data using scalable Azure resources.Utilized advanced Snowflake features such as Common Table Expressions (CTE), Time Travel, and Zero Copy Cloning for efficient data management and querying.Utilized PySpark and Spark-SQL for data extraction, transformation, and aggregation from multiple file formats, ensuring efficient and scalable data processing.Led the design and implementation of Azure Data Catalog solutions to improve data discovery, governance, and metadata management throughout the organization.End-to-end implementation project experience in data processing pipeline using Azure Data Lake, Azure Synapse Analytics, and Azure Data Factory.Developed robust data warehousing solutions using Azure Data Factory, ensuring seamless integration with multiple data sources.Oversaw the management and organization of metadata using Azure Data Catalog tools, ensuring consistent and accurate data descriptions, lineage, and classification to enhance data governance.Designed and optimized complex SQL queries, stored procedures, and functions to support data extraction, transformation, and loading (ETL) processes in Azure Synapse Analytics.Developed and maintained data pipelines and data warehousing solutions on Snowflake, enabling efficient data storage and retrieval.Designed and implemented robust data warehouse solutions using Azure Synapse Analytics, enabling efficient data storage, retrieval, and analysis.Conducted training sessions for team members on best practices and advanced features of Azure Data Factory and Azure Databricks.Created and maintained detailed technical specification documents for Snowflake builds.Created complex integration workflows using Azure Data Factory to automate data transfer and transformation processes across various enterprise systems.Implemented performance-tuning techniques for SQL queries, including indexing, query optimization, and partitioning to enhance database performance.Utilized Power BI for creating and managing detailed reports, dashboards, and data visualizations, enabling data-driven decision-making.Implemented data security and governance policies in Snowflake and Azure Synapse Analytics to ensure compliance with industry standards and regulations.Utilized Snowflakes features such as virtual warehouses, clustering, and data sharing to optimize performance and cost management.Conducted performance tuning and query optimization in Snowflake, improving query response times by up to 40%.Integrated Snowflake with Azure Data Factory, BI platforms, and third-party applications to create a unified data ecosystem. Experienced in development methodologies like Agile/Scrum.Extensive experience in designing, developing, documenting, and testing ETL jobs and mappings using Azure Data Factory and Azure Databricks to populate tables in Data Warehouses and Data marts.Expertise in transforming business requirements into analytical models, designing algorithms, building models, developing Data Mining, Data Acquisition, Data Preparation, Data Manipulation, Feature Engineering, Machine Learning Algorithms, Validation and Visualization, and reporting solutions that scale across massive volumes of structured and unstructured data.Designed and developed ETL processes using Azure Data Factory to extract, transform, and load data from various sources into data warehouses.Optimized ETL processes in Azure Data Factory for performance and scalability, reducing execution times by up to 30%.Developed and maintained PL/SQL scripts and packages for complex data manipulations and business logic implementation in Azure Synapse Analytics.Conducted training sessions for team members on best practices and advanced features of Azure Data Factory and Azure Databricks, enhancing team capabilities and project efficiency.Created interactive and insightful dashboards in Power BI, leveraging advanced visual analytics to present data effectively.Developed data pipelines to synchronize and aggregate data from various data sources using Azure Data Factory. Built the query data platform and provided guidelines for partner teams to use aggregated data, publishing standard data sets and schemas.Experience with source control systems such as Git, Bitbucket, and Jenkins. Working experience in CI/CD deployments.Implemented CI/CD pipelines using DevOps tools such as Jenkins, GitLab CI/CD, and Azure DevOps to automate application deployment and testing.TECHNICAL SKILLS:Big Data TechnologiesHadoop, MapReduce, Spark, HDFS, Sqoop, YARN, Oozie, Hive, Impala, Zookeeper, Apache Flume, Apache Airflow, Cloudera, HBaseProgramming LanguagesPython, PL/SQL, SQL, Scala T-SQL,Cloud ServicesAzure Data Lake Storage Gen 2, Azure Data Factory, Blob storage, Azure SQL DB, Databricks, Azure Event Hubs, Delta Tables, Cloud functions, Clusters.DatabasesMySQL, MS Access, Teradata, and SnowflakeNoSQL Data BasesMongoDB, Cassandra DB, HBaseDevelopment StrategiesAgile, Lean-Agile, Pair Programming, Waterfall, and Test-Driven Development.Visualization & ETL toolsTableau, Informatica, Power BI,Version control ToolsJenkins, Git, and SVNOperating SystemsUnix, Linux, Windows, Mac OSRole: Data EngineerClient: SMBCDuration: Jan 2024 PresentResponsibilities:Developed Spark applications using PySpark and Spark-SQL on Azure Databricks for data extraction, transformation, and aggregation from multiple file formats.Developed Spark streaming applications to read raw packet data from Kafka topics, format it to JSON, and push it back to Kafka for future use cases.Collected data using Spark Streaming from Azure Data Lake Storage in near-real-time, performing necessary transformations and aggregations on the fly to build a common learner data model and persist the data.Migrated on-premises data warehouses to Azure Synapse Analytics, reducing infrastructure costs and improving scalability.Implemented data partitioning and clustering strategies in Azure Synapse to enhance query performance.Demonstrated proficiency in using Azure Data Catalog for effective metadata management and data governance.Gained experience and knowledge of real-time data analytics using Spark Streaming, Kafka, and Flume on Azure.Prepared scripts to automate the ingestion process using PySpark and Scala through various sources such as API, Azure Blob Storage, Teradata, and Azure SQL Database.Created SQL scripts for data extraction, transformation, and loading (ETL) processes, ensuring efficient and accurate data movement across different environments.Developed reusable components and templates in Azure Data Factory to standardize ETL processes and improve development efficiency.Automated ETL job scheduling and monitoring in Azure Data Factory to ensure timely and reliable data processing.Utilized Azure Data Factory to perform data transformation tasks, including data aggregation, filtering, sorting, and data validation, ensuring data integrity and accuracy.Migrated legacy ETL processes to Azure Data Factory, improving maintainability and performance.Enabled data orchestration and workflow automation using Informatica IICS and Control M.Designed and developed ETL processes in Azure Data Factory to migrate campaign data from external sources into Azure Synapse Analytics.Implemented installation and configuration of a multi-node cluster on Cloud using Azure Virtual Machines.Collected data using Spark Streaming from Azure Data Lake Storage in near-real-time and performed necessary transformations and aggregations.Designed, implemented, and monitored ETL processes to ensure efficient data flow from various sources into data warehouses on Azure.Conducted regular data analysis to generate actionable insights, supporting business strategies and operations.Integrated Azure Cosmos DB using Azure Functions to store the values of items and backup the Cosmos DB streams.Conducted regular audits of metadata entries in the Azure Data Catalog to ensure accuracy and reliability, identifying and rectifying any discrepancies or errors.Designed and developed ETL processes in Azure Data Factory to migrate campaign data from external sources like Azure Blob Storage, ORC/Parquet/Text Files into Azure Synapse Analytics.Stored log files in Azure Blob Storage, using versioning for highly sensitive information.Developed streaming pipelines using Azure Event Hubs and Azure Stream Analytics to analyze data for dealer efficiency and open table counts for data coming from IoT-enabled poker and other pit tables.Environment: Python, Spark, DataStage, Azure Virtual Machines, Azure Data Lake Storage, Azure Synapse Analytics, Azure Data Factory, Azure Blob Storage, SQL, Tableau, Docker, Git, REST, Bitbucket, Jira.Role: Data EngineerClient: ExperianDuration: Oct 2021 Dec 2023Responsibilities:Designed complex ETL mappings and transformations in Azure Data Factory to support data warehousing and business intelligence initiatives.Collaborated with business analysts to gather and refine ETL requirements, ensuring alignment with business objectives.Conducted impact analysis and data profiling using Azure tools to ensure data quality and consistency before ETL development.Performed detailed data quality and consistency checks within the Azure Data Catalog, ensuring that all metadata entries are accurate, complete, and up to date.Implemented Azure Virtual Machines, Key Vault, Network Security Groups, Auto Scaling, Load Balancers, and Azure Functions using Azure API and exposed them as Restful Web services.Processed and analyzed large datasets using Azure Databricks and Parquet files to support data-driven decision-making.Utilized PySpark and Spark-SQL for data extraction, transformation, and aggregation from multiple file formats, ensuring efficient and scalable data processing.Proficient in Azure Data Factory, Azure Data Lake Storage, and Azure Synapse Analytics for data integration, storage, and processing.Leveraged Snowflakes data sharing capabilities to facilitate seamless data access across different teams and departments.Created Spark streaming applications to read raw packet data from Kafka topics, format it to JSON, and push it back to Kafka for future use cases, supporting real-time data processing.Involved in creating and managing Azure Synapse Analytics tables, loading data, and writing SQL queries for data processing.Developed custom stored procedures and user-defined functions in Azure Synapse Analytics to enhance data processing capabilities.Conducted impact analysis and data profiling using Azure Databricks to ensure data quality and consistency before ETL development.Processed and analyzed large datasets using Azure Databricks and Parquet files to support data-driven decision-making.Implemented robust error handling and exception management in SQL programs to ensure smooth execution and effective error logging.Utilized Azure Synapse Analytics' time travel feature to recover and analyze historical data for audit and compliance purposes.Implemented data masking and encryption in Azure Synapse Analytics to protect sensitive information.Automated complex workflows and batch processes using Control M to enhance operational efficiency.Designed and implemented data models and schemas in Snowflake to support various business intelligence and reporting needs.Designed and developed ETL workflows using Azure Data Factory and Azure Databricks for processing data in Azure Data Lake Storage.Developed SQL scripts for data extraction, transformation, and loading (ETL) processes, ensuring efficient data movement.Implemented continuous integration and continuous deployment (CI/CD) best practices using Azure DevOps to streamline development workflows.Created insightful dashboards and reports using Power BI and Azure Data Catalog analytics features, providing valuable insights into data usage patterns, data quality issues, and metadata trends.Automated infrastructure provisioning and management using cloud-native DevOps tools like Azure Resource Manager and Terraform.Developed and maintained custom scripts and automation tools using Azure DevOps to enhance workflows and processes.Exported analyzed data to relational databases using Azure Data Factory for visualization and generated reports for the BI team using Power BI.Environment: Azure Data Factory, Azure Synapse Analytics, Azure Data Lake Storage, Azure Virtual Machines, Azure Key Vault, Azure DevOps, Power BI, Databricks, SQL, HDFS, MapReduce, Snowflake, CloudWatch, Hive, Sqoop, Pig, HBase, Apache Spark, Oozie Scheduler, Java, Python, Scala, Teradata, Netezza, Oracle.Role: pythonClient: Enterprise Software Solutions, IndiaDuration: June 2019 -May 2021Responsibilities:Designed the front end and back end of the application utilizing Python on Django Web Framework.For the development of the user interface of the website using HTML, CSS, and JavaScript.Developed Business Logic using Python on the Django Web Framework.Developed complex web, middleware, and back-end systems in Python, SQL, and Perl for Linux and Windows.Created very interactive and insightful dashboards in Tableau leveraging advanced visual analytics.Experience in developing views and templates with Python and Django's view controller and templating language to create a user-friendly website interface.Established data quality monitoring frameworks within the Data Catalog to ensure data accuracy, completeness, and consistency across the data platform.Provided technical support and troubleshooting for production DataStage jobs, ensuring data processing continuity.Participated in code review sessions and provided constructive feedback to improve the quality of DataStage ETL solutions. Analyzing the requirements. Developing code to run the application.Committing the code in Bit bucket for code tracking.Deploying the code into Server and testing.Used SQL Loader to load data from flat files received from various facilities every day.Implemented data masking and encryption in Snowflake to protect sensitive information.Participated in architecture design sessions to define Snowflake implementation strategies and best practices.Provided ongoing support and troubleshooting for Snowflake environments, ensuring optimal performance and uptime.Created interactive dashboards in Tableau leveraging advanced visual analyticsExperience in Dynamic SQL, Collections, and Exception Handling.Worked with the ETL developers to get data from various data sources (SQL Server, flat files & XML) Experience with performance tuning of Oracle database using Explain Plan and Hints to perform cost-based optimization.Utilized standard Python modules such as CSV, robot parser, iterators, and pickle for development.Environment: Python, Django, CSS, DataStage, snowflake, python libraries NumPy, SQL Alchemy, MySQL DB, Jenkins, |
