| | | 20,000+ Fresh Resumes Monthly | |
|
|
| |
Click here or scroll down to respond to this candidateCandidate's Name
EMAIL AVAILABLEPHONE NUMBER AVAILABLE, PHONE NUMBER AVAILABLESenior Data EngineerLinkedIn: http://LINKEDIN LINK AVAILABLEPROFILE SUMMARYWith over 10+ years of data engineering expertise, proficient in both cloud and on-premise data solution architectures.Proficient in designing and deploying scalable data pipelines using Apache Spark, enhancing processing efficiency and speed.Experienced in managing and optimizing AWS Redshift data warehouses, ensuring robust performance and scalability.Skilled in developing ETL processes with Apache Kafka and AWS Glue, facilitating real-time and batch data streaming.Expert in SQL and Python scripting, providing complex data analysis and automation solutions across various projects.Utilized Docker and Kubernetes extensively for containerization and orchestration, improving deployment processes and environment consistency.Implemented data ingestion and transformation using AWS S3 and Azure Data Lake Storage (ADLS) to support diverse data operations.Designed and maintained efficient data models and databases using DynamoDB and Oracle Database, ensuring high availability and performance.Developed predictive models with TensorFlow, applying machine learning to predict trends and enhance decision-making.Proficient with Jenkins for continuous integration and deployment, automating workflow pipelines to increase development efficiency.Leveraged Informatica PowerCenter for robust data integration and transformation, enhancing data connectivity and flow.Managed extensive data warehousing solutions using Azure SQL Data Warehouse, optimizing data storage and access.Executed data migration and integration projects using Talend, ensuring accurate and efficient data transfer between systems.Developed and maintained real-time analytics platforms using Apache Beam and Apache Hadoop, handling large-scale data efficiently.Utilized PostgreSQL and PL/SQL for database management and complex query execution, enhancing data retrieval and management.Experienced in data visualization and business intelligence using Tableau and QlikView, creating dynamic and insightful dashboards.Applied GitLab and GIT for version control, maintaining code integrity and supporting collaborative project environments.Architected and maintained scalable cloud environments using AWS EMR and Azure Cosmos DB, supporting big data technologies.Designed data lakes with AWS Lake Formation, centralizing data management and enhancing security and compliance.Utilized Scala for backend data processing scripts, improving efficiency and performance in data applications.Implemented robust data security measures using Azure and AWS cloud services, ensuring compliance with industry regulations.Applied Agile methodologies in data project management, promoting rapid development cycles and adaptive planning.Leveraged Apache Hive for data warehousing and analysis, optimizing data queries and improving data warehouse structure.Developed and optimized APIs using Python (pandas, NumPy) for data manipulation and integration across various platforms.Enhanced system performance and data processing capabilities using Apache Spark and Terraform for infrastructure management.Conducted advanced data analysis and processing using R, supporting complex statistical analysis and data interpretation.TECHNICAL SKILLSProgramming Languages : SQL, Python, Scala, PL/SQL, RBig Data Technologies : Apache Hadoop, Apache Spark, Apache Kafka, Apache HiveData Storage : AWS S3, Azure Data Lake Storage, DynamoDB, PostgreSQL, Oracle DatabaseETL Tools : Informatica PowerCenter, Talend, AWS Data Pipeline, Azure Data FactoryData Warehousing : AWS Redshift, Azure SQL Data WarehouseCloud Platforms : AWS, AzureMachine Learning : TensorFlowContainerization : Docker, KubernetesInfrastructure as Code : TerraformVersion Control : GIT, GitLabData Visualization : Tableau, QlikViewStreaming Technologies : Apache KafkaAutomation/CI/CD : JenkinsDatabase Management : DynamoDB, PostgreSQL, Oracle DatabaseProject Management : Agile methodologiesWORK EXPERIENCE:Global Atlantic financial group, Indianapolis, IN Nov 2022- Till NowSr. Data EngineerRoles & Responsibilities:Designed and implemented large-scale data processing pipelines utilizing Apache Spark to enhance analytics capabilities in the insurance sector.Managed and optimized data storage and retrieval processes using AWS Redshift, focusing on performance improvements and cost efficiency.Developed robust data ingestion mechanisms with AWS S3, employing ETL best practices to ensure high data quality and accessibility.Implemented real-time data streaming solutions with Apache Kafka, facilitating immediate data availability and processing.Constructed and maintained scalable and efficient data models in DynamoDB to support high-throughput applications and data consistency.Utilized Kubernetes for the deployment and management of containerized applications, enhancing operational scalability and reliability.Performed complex data querying and manipulation tasks using SQL to support critical business decision-making processes.Developed predictive analytics models using TensorFlow, effectively forecasting trends and behaviors in the insurance market.Architected and maintained a centralized data lake using AWS Lake Formation, simplifying data access and analysis.Enhanced data security and compliance measures using AWS-managed services to protect sensitive insurance information.Optimized data transformation processes with DBT, increasing data processing efficiency and reducing operational costs.Tailored data retrieval and storage strategies using Python scripting, enhancing system performance in data-intensive applications.Designed robust data pipelines to support both data streaming and batch processing, handling diverse data workloads efficiently.Integrated agile project management methodologies into data operations to enhance project flexibility and stakeholder satisfaction.Monitored, diagnosed, and resolved issues within data pipelines, ensuring high availability and reliability of data services.Executed data archiving strategies to manage the lifecycle of vast amounts of insurance data efficiently.Enhanced data accessibility and interoperability across different departments within the insurance company, promoting data-driven decisions.Conducted thorough code reviews and optimizations to improve performance and maintainability of data systems.Led data governance initiatives to ensure high standards of data quality and consistency across all platforms.Developed custom data connectors to facilitate seamless integration with external data sources, expanding data capabilities.Trained team members on modern data management practices and technologies, fostering a culture of continuous improvement.Automated data pipeline deployments using continuous integration and deployment practices with GIT, streamlining workflow processes.Designed stringent data security frameworks to safeguard sensitive insurance information against potential cyber threats.Spearheaded the adoption of new technologies and tools in the data engineering field to maintain industry competitiveness.Collaborated with business stakeholders to align data solutions with strategic objectives, ensuring optimal business impact.Environment: Apache Spark, AWS Redshift, AWS S3, Apache Kafka, DynamoDB, Kubernetes, SQL, TensorFlow, AWS Lake Formation, AWS-managed services, DBT, Python, GIT.First Republic Bank, New York, NY Feb 2020 Jan 2022Title: Data EngineerRoles & Responsibilities:Designed scalable data processing pipelines using Apache Spark, significantly improving data handling capabilities in the financial services sector.Managed and optimized AWS Redshift, enhancing data warehousing solutions for improved data analysis and storage efficiency.Developed data ingestion pipelines with AWS S3 and Data Pipeline, ensuring reliable data transfer and storage for critical financial operations.Implemented real-time data streaming applications using Apache Kafka, boosting responsiveness and data flow in transaction processing.Utilized Kubernetes for orchestrating containerized applications, enhancing deployment processes and operational reliability in cloud environments.Employed Apache Beam for building robust data processing jobs, facilitating efficient data manipulation and aggregation.Performed complex data transformations using AWS Glue, automating ETL processes and reducing manual intervention.Configured DynamoDB for high-performance data storage solutions, supporting dynamic data access needs in real-time applications.Developed and maintained SQL scripts for data querying and manipulation, aiding in complex financial data analysis.Leveraged AWS EMR for managing big data frameworks, optimizing data processing tasks across distributed computing environments.Implemented TensorFlow for developing advanced machine learning models, aiding in predictive analytics for financial forecasting.Architected AWS Lake Formation to streamline data lake management, enhancing data discovery and security.Utilized Scala for backend data processing tasks, improving system efficiency and performance in data operations.Applied GIT for version control, ensuring code integrity and supporting collaborative development efforts.Integrated Agile methodologies into project management, enhancing team collaboration and project delivery timelines.Developed data pipelines using AWS Data Pipeline, automating data workflows and improving operational efficiency.Executed data transformations and integrations using SBT, standardizing data formats and improving data quality.Configured AWS Redshift for advanced data warehousing, enabling sophisticated query execution and data management.Utilized Python for scripting and automation, enhancing data processing capabilities and operational efficiency.Employed AWS Glue for data cataloging and ETL operations, facilitating seamless data integration and management.Architected secure data solutions using DynamoDB and AWS Lake Formation, ensuring data integrity and compliance.Designed and executed data archiving strategies with AWS S3, effectively managing data lifecycle and storage costs.Led data governance initiatives, establishing standards and practices to ensure data quality and consistency across all platforms.Environment: Apache Spark, AWS Redshift, AWS S3, Apache Kafka, Kubernetes, Apache Beam, AWS Glue, DynamoDB, SQL, AWS EMR, TensorFlow, AWS Lake Formation, Scala, Git, Agile methodologies, AWS Data Pipeline, SBT, Python, and AWS S3.Mr. Cooper, Irving, TX Nov 2017- Jan 2020Title: Big Data EngineerRoles & Responsibilities:Designed and implemented data processing pipelines using Apache Hadoop, significantly enhancing data handling capabilities for loan processing.Leveraged Apache Spark for complex data processing tasks, improving performance and scalability of data analytics operations.Configured and managed Azure Data Lake Storage (ADLS) to store and process large volumes of loan application data efficiently.Utilized Azure Data Factory (ADF) for orchestrating and automating data movement and transformation, streamlining ETL processes.Implemented real-time data ingestion and processing using Apache Kafka, enabling immediate data availability for decision-making.Employed Docker for containerizing applications, enhancing deployment processes and environment consistency across development and production.Managed Terraform scripts for provisioning and managing infrastructure as code, ensuring consistent environments across multiple deployments.Developed and maintained Azure Cosmos DB for high throughput and low latency data access, supporting dynamic query requirements.Utilized Azure SQL Database for managing structured data, enhancing data accessibility and reliability for critical financial operations.Architected and maintained an Azure SQL Data Warehouse, optimizing data storage and complex querying capabilities for financial analytics.Developed ETL pipelines using Talend, automating data transformation and loading, thus improving data quality and processing time.Scripted complex data manipulation tasks using Python, enhancing automation and efficiency in data processing workflows.Employed SQL extensively for data querying and manipulation, supporting critical business reporting and analytics functions.Configured Apache Kafka for managing data streams, facilitating real-time data processing and integration.Implemented Docker containers to encapsulate application environments, ensuring consistent operations across different computing environments.Utilized Kubernetes for orchestrating container deployment, scaling, and management, improving operational agility and system resilience.Automated infrastructure management using Terraform, enhancing infrastructure provisioning and deployment processes.Developed and optimized data models in Azure Cosmos DB, ensuring optimal performance for query execution and data retrieval.Orchestrated data pipeline workflows using Azure Data Factory, enhancing automation and integration of data processing tasks.Managed the lifecycle of big data solutions, from conception through deployment, ensuring alignment with business goals andtechnical requirements.Environment: Apache Hadoop, Apache Spark, Azure Data Lake Storage (ADLS), Azure Data Factory (ADF), Apache Kafka, Docker, Terraform, Azure Cosmos DB, Azure SQL Database, Azure SQL Data Warehouse, Talend, Python, SQL, Kubernetes.Sun Technologies, India Aug 2016 - Sep 2017Title: Data AnalystRoles & Responsibilities:Utilized Microsoft Excel for complex data analysis, creating pivot tables and charts to support business decisions.Managed PostgreSQL databases, optimizing data storage and retrieval processes for enhanced performance and efficiency.Developed scripts using Python, including libraries like pandas and NumPy, to automate data cleaning and manipulation tasks.Configured Apache Hive to manage and query large data sets, improving data accessibility for analytics purposes.Created dynamic dashboards and reports using Tableau, providing actionable insights to business stakeholders.Employed GitLab for version control, maintaining code integrity and facilitating collaborative development efforts.Integrated data from various sources into PostgreSQL, ensuring data consistency and accuracy for analysis.Designed and maintained data warehouses using PostgreSQL, enhancing data aggregation and summarization capabilities.Developed and automated ETL processes, utilizing Python to transform and load data efficiently into data warehouses.Implemented QlikView for developing interactive visualizations, enhancing data interpretation and decision-making processes.Scripted data extraction and loading tasks using Python, streamlining data updates and maintenance.Managed data documentation, ensuring all processes and datasets were well-documented for compliance and reference.Utilized SQL extensively to perform complex queries, supporting analytical tasks and generating insights from data.Collaborated with business teams to identify data requirements and deliver tailored analytical solutions.Conducted data quality checks using Python, ensuring the accuracy and reliability of analytics outputs.Optimized data storage solutions in PostgreSQL, improving system performance and data retrieval speeds.Leveraged Apache Hive for big data analytics, enabling efficient handling of large datasets across distributed systems.Trained junior analysts on data tools and best practices, enhancing team capabilities and productivity.Environment: Microsoft Excel, PostgreSQL, Python (pandas, NumPy), Apache Hive, Tableau, GitLab, QlikView, SQL, Data warehouses (PostgreSQL), ETL (Python), Apache Hive.MLM Software, Gurgaon, Haryana May 2014- Jun 2016Title: SQL DeveloperRoles & Responsibilities:Developed and maintained SQL databases using Oracle Database, enhancing data storage and retrieval operations.Wrote complex PL/SQL scripts for data manipulation and business logic implementation, optimizing database functionality.Configured and managed Informatica PowerCenter for ETL processes, improving data integration and workflow efficiency.Utilized Apache Kafka for real-time data streaming, facilitating timely data availability for transaction processing.Created comprehensive data visualizations and dashboards using Tableau, supporting strategic business decision-making.Employed Jenkins for automating builds and deployment processes, enhancing code integration and delivery.Implemented Docker containers to encapsulate the development environment, ensuring consistency across multiple platforms.Conducted data analysis and reporting using R, providing insights into customer behaviors and market trends.Developed analytics solutions with R, enabling sophisticated statistical analysis and predictive modeling.Managed database schemas and performed database tuning using Oracle tools, ensuring optimal performance.Scripted automated data import/export tasks in PL/SQL, enhancing data interchange between systems.Collaborated with data teams to define requirements and design database solutions tailored to business needs.Maintained data integrity and security within Oracle databases, implementing best practices and compliance measures.Trained new team members on Oracle Database management and PL/SQL programming, enhancing team skills.Optimized existing SQL queries and procedures, improving performance and reducing processing time.Environments: Oracle Database, PL/SQL, Informatica PowerCenter, Apache Kafka, Tableau, Jenkins, Docker, R.HIGHER EDUCATION:Bachelors specialization in Computer Science from JNTUH University, INDIA. (2010-2014). |
