| | | 20,000+ Fresh Resumes Monthly | |
|
|
| |
Click here or scroll down to respond to this candidate Name: - Candidate's Name
PHONE NUMBER AVAILABLE
Linked In: - https://LINKEDIN LINK AVAILABLE
EMAIL AVAILABLE
Senior Data Engineer
PROFESSIONAL SUMMARY:
Over 8+ years of experience as a Senior Data Engineer, specializing in data pipeline architecture, optimization, and
data engineering across various industries.
At global investment management firms, utilized Apache Kafka, AWS S3, and Databricks to design and optimize large-
scale data pipelines for real-time data processing.
Developed robust ETL pipelines with Apache Airflow, AWS Glue, and DBT for investment management companies,
ensuring smooth data ingestion and transformation processes.
At Technology services companies leveraged AWS Glue, Informatica, and Apache NiFi to implement scalable data
solutions focused on data integration and cloud-based processing.
For multinational retail corporations, built and optimized data platforms using Azure Cosmos DB, Azure Data Lake
Storage (ADLS), and Azure HDInsight to enhance data management and real-time analytics.
Proficient in programming languages such as Python, Scala, and SQL for developing data processing solutions across a
range of industries.
Expertise with relational and NoSQL databases including MySQL, PostgreSQL, MongoDB, Snowflake, and AWS
DynamoDB for high-performance data storage and retrieval.
Designed and optimized ETL pipelines using tools like Talend, Apache Airflow, and AWS Glue, ensuring efficient data
flows for businesses.
Experience with cloud platforms, particularly AWS S3, EMR, DynamoDB, Glue, and Azure services like Cosmos DB and
ADLS, enabling scalable cloud data solutions.
Built and maintained big data solutions using Apache Kafka, Databricks, and Snowflake for large-scale data processing
in retail and finance.
Proficient in data visualization tools such as Power BI, Tableau, and QlikView, creating interactive dashboards for
business insights.
Developed machine learning workflows using PyTorch and Kubeflow for predictive analytics and modeling in
investment management.
Expertise in Terraform for automating infrastructure deployment and management across cloud environments.
Led CI/CD pipelines using Jenkins and Docker, ensuring smooth development and deployment workflows.
Skilled in data governance and security, implementing compliance measures using AWS IAM, Azure Active Directory,
AWS KMS, and Azure Key Vault.
Hands-on experience with Apache Kafka and Azure Stream Analytics for real-time data processing in retail and
technology services sectors.
Proficient in data APIs using Flask and FastAPI, enhancing data accessibility and integration across platforms.
Applied Agile project management methodologies using JIRA, enhancing project efficiency across industries.
Strong background in big data ecosystems including Apache Hadoop, Apache Hive, and Azure HDInsight, streamlining
large dataset processing.
Expertise in data modeling and management using tools like DBT and Pandas, improving data quality for analytics.
Implemented operational analytics using AWS QuickSight and Power BI to deliver real-time insights and reporting
solutions.
Hands-on experience with data encryption and security measures using AWS KMS and Azure Key Vault for data
integrity.
Successfully performed legacy system data migrations using Apache NiFi and Informatica, modernizing legacy
infrastructure across different industries.
Enhanced data pipeline performance through optimization techniques, reducing latency and improving data
throughput.
Developed and maintained real-time data streaming solutions using Apache Kafka and Azure Stream Analytics for
immediate insights and decision-making.
TECHNICAL SKILLS:
Programming Languages: Python, Scala, SQL, T-SQL
Databases: MySQL, PostgreSQL, MongoDB, Snowflake, AWS DynamoDB, Azure SQL Database, Teradata, SQL Server,
Click House
Data Processing & ETL Tools: Talend, Apache Airflow, Informatica, Apache NiFi, AWS Glue, Apache Hadoop, Apache
Hive, Matillion, SSIS, Azure Data Factory (ADF)
Cloud Platforms: AWS (S3, EC2, EMR, DynamoDB, Glue, Redshift,Lambda), Azure (Cosmos DB, HDInsight, ADLS,
Stream Analytics, Azure Databricks), Google Cloud Platform
Data Visualization: Power BI, Tableau, QlikView, AWS QuickSight
Big Data Technologies: Apache Kafka, Databricks, Apache Spark, Snowflake schema
Machine Learning: PyTorch, Kubeflow, Databricks
Infrastructure as Code: Terraform, AWS CloudFormation
Continuous Integration/Deployment: Jenkins, Docker,Kubernet
Security & Compliance: AWS IAM, Azure Active Directory, AWS KMS, Azure Key Vault
Workflow Orchestration: Apache Airflow, Apache NiFi
Real-Time Data Processing: Apache Kafka, Azure Stream Analytics
Version Control: Subversion (SVN),Git
API Development: Flask, FastAPI
Agile Project Management: JIRA, Agile methodologies
Certifications: DP-203 Azure Data Engineer Associate
PROFESSIONAL EXPERIENCE:
Client: The Vanguard Group, Valley Forge, PA Jan 2023 to till date
Role: Senior Data Engineer
Roles & Responsibilities:
Developed robust data pipelines using Apache Kafka, AWS EMR, and Apache Airflow, streamlining and
enhancing data processing efficiency.
Optimized data storage with MongoDB and AWS S3, ensuring fast and reliable data retrieval.
Designed and managed scalable ETL pipelines leveraging Airflow and Databricks, enhancing data handling and
transformation capabilities.
Integrated Python, SQL, and DBT for collaborative analytics, improving reporting and insights.
Enabled real-time data streaming with Apache Kafka and AWS EMR, ensuring timely data availability for financial
operations.
Utilized Pandas in Python for advanced data manipulation and financial analysis, improving data accuracy.
Configured and deployed machine learning workflows with Kubeflow and AWS Lambda, enhancing predictive
analytics and event-driven tasks.
Automated routine data workflows using AWS Glue, AWS Lambda, and Python, boosting operational efficiency.
Enhanced query performance with SQL and Databricks, reducing response times for financial queries.
Developed APIs using Python and Flask, ensuring efficient data exchanges between systems.
Deployed scalable and secure infrastructure using Kubernetes, Terraform, and Git across cloud environments,
enhancing deployment processes.
Managed CI/CD pipelines with Jenkins, streamlining software updates, builds, and maintenance.
Utilized Git for version control, ensuring code integrity and managing changes effectively across development
projects.
Improved data security and compliance with AWS tools, and automated backups using AWS S3.
Led data migrations to AWS S3 and DynamoDB, ensuring scalability, security, and seamless transitions.
Enhanced data visualization and financial trend insights using AWS QuickSight dashboards, enabling better
business intelligence.
Integrated legacy systems with real-time data streams using Apache Kafka, AWS Lambda, and AWS EMR,
improving overall system efficiency.
Provided training on Databricks, Python, Kubernetes, and AWS Lambda, enabling team growth and adoption of
new technologies.
Conducted data quality checks and ensured high standards of accuracy with Python scripts, ensuring data
integrity.
Environment: Apache Kafka, AWS EMR, Apache Airflow, MongoDB, AWS S3, AWS Glue, Databricks, Python, SQL, DBT,
Pandas, Kubeflow, Teradata, AWS Lambda, AWS QuickSight, Jenkins, Terraform, Flask, Kubernetes, Git.
Client: Creamos Solutions Inc, Fremont, CA May 2021 to Jun 2022
Role: Data Engineer
Roles & Responsibilities:
Led data integration and ETL processes using AWS Glue, optimizing data management and improving workflow
efficiency.
Enhanced storage and scalability with AWS S3, DynamoDB, AWS EC2, and Terraform, providing robust and flexible
data solutions.
Implemented CI/CD workflows and Agile practices, improving project timelines and ensuring smooth delivery of
software.
Managed data transformations using Python and AWS technologies, maintaining high data quality and consistency
across systems.
Developed data analytics and visualizations with Tableau, improving business intelligence and data-driven
decision-making.
Maintained scalable data pipelines with Python, SQL, and modern ETL methods, ensuring reliable data flow and
processing.
Deployed containerized applications with Docker, improving scalability and system management.
Integrated machine learning models using PyTorch, enabling advanced predictive analytics for deeper insights.
Automated data workflows using AWS Glue and Apache NiFi, reducing manual intervention and increasing
operational efficiency.
Optimized real-time data processing using Apache Kafka and ClickHouse, ensuring immediate data availability and
system responsiveness.
Managed cloud data migrations and disaster recovery plans using Terraform and AWS EC2, ensuring data integrity,
system resilience, and business continuity.
Developed APIs using Flask, enabling seamless data exchange between different systems.
Conducted data quality assessments using Python scripts, ensuring high standards of accuracy and reliability in
data operations.
Engineered microservices using Docker, enhancing modular application deployment and flexibility.
Implemented Agile project management techniques, aligning development processes with business goals and
improving team adaptability.
Monitored system performance using AWS CloudWatch and AWS EC2, ensuring optimal operations and quick
issue resolution.
Provided training on Tableau and Docker, increasing team capabilities and ensuring effective technology adoption.
Configured and automated project tracking and workflow management using JIRA, improving team collaboration
and project visibility.
Environment: AWS Glue, AWS S3, AWS EC2, Docker, Terraform, AWS DynamoDB, Agile, Python, Tableau, SQL, JIRA, Flask,
PyTorch, AWS IAM, Apache Kafka, ClickHouse, AWS CloudWatch.
Client: Walmart, Bentonville, AR Jul 2018 to Apr 2021
Role: Cloud Engineer
Roles & Responsibilities:
Managed data migration projects to Azure Cosmos DB and Azure Data Lake Storage (ADLS), ensuring data integrity and
availability.
Implemented big data analytics solutions with Azure HDInsight, Azure Databricks, and Azure Data Factory (ADF) to
enhance data insights and streamline ETL processes.
Developed real-time analytics using Azure Stream Analytics and PostgreSQL, improving operational decision-making
capabilities.
Engineered infrastructure as code with Terraform, automating cloud deployments and managing resources efficiently.
Optimized data processing with Apache Hadoop, Kafka, and Snowflake schema, enhancing system performance and
throughput.
Integrated Snowflake for scalable data warehousing, supporting complex data analysis and storage needs.
Configured data processing pipelines using Apache Kafka, facilitating dynamic data flow and batch processing.
Utilized PostgreSQL and Azure SQL Database for advanced query operations, enhancing data retrieval and supporting
analytics.
Maintained ETL pipelines with Scala and SQL in Databricks, streamlining data transformations and loading.
Automated infrastructure provisioning with Terraform, ensuring consistent deployments and reducing manual effort.
Implemented data lake solutions with Azure ADLS, optimizing storage and management for large-scale data.
Monitored data jobs and system health with Azure Monitor, ensuring high performance and availability.
Tuned data processing applications in Apache Hadoop to boost efficiency and reduce processing times.
Created dashboards and visualizations in Power BI, providing real-time insights into business operations.
Leveraged Azure Stream Analytics for real-time analytical solutions, enabling quick responses to market changes.
Environment: Azure Cosmos DB, Azure Data Lake Storage (ADLS), Azure HDInsight, Azure Databricks, Azure Stream Analytics,
PostgreSQL, Azure SQL Database, Terraform, Apache Hadoop, Kafka, Snowflake, Scala, Apache Spark, Azure Monitor, Power BI,
Azure Data Factory (ADF).
Client: Tridhya Tech, Ahmedabad, India Jun 2017 to Apr 2018
Role: Data Quality Analyst
Roles& Responsibilities:
Analyzed and enhanced data quality using SQL and Python, ensuring high data integrity across platforms.
Developed data visualizations and dashboards with QlikView, delivering actionable insights for business decisions.
Optimized data warehousing and processing with Apache Hadoop, Hive, and PySpark, improving query execution and
data analysis.
Automated reporting tasks and data transformations using Python and PySpark, increasing operational efficiency.
Conducted data quality audits using SQL and PySpark, identifying and resolving anomalies to maintain high standards.
Configured and tuned Apache Hive to enhance the performance of large-scale data queries.
Developed scalable data models in Hadoop and PySpark to support complex business intelligence needs.
Enhanced team collaboration through Agile methodologies, improving project delivery and product quality.
Provided technical training on data analytics tools and practices, boosting team proficiency.
Utilized Unix for managing and processing large datasets, improving workflow efficiency.
Environment: SQL, Python, PySpark, QlikView, Apache Hadoop, Apache Hive, Subversion (SVN), Agile methodologies,UNIX
Client: Yana Software Private Limited, Hyderabad, India Jan 2016 to May 2017
Role: SQL Developer
Roles & Responsibilities:
Developed and optimized complex SQL queries, stored procedures, views, and indexes using DML, DDL commands,
and user-defined functions to support business logic.
Enhanced database performance through query optimization and execution plan analysis.
Implemented normalization and de-normalization of tables to improve query efficiency and results.
Created and managed SSIS packages for dynamic database modifications and data integration.
Designed and generated diverse reports (matrix, tabular, chart) using SSRS to support business reporting needs.
Migrated on-premise databases to SQL Server 2014, coordinating with teams for smooth transition and updates.
Managed multiple SQL Server 2012 databases, adapting to evolving business requirements.
Collaborated with teams and stakeholders to clarify tasks, address issues, and improve systems.
Environment: SQL Server Management Studio, Microsoft Visual Studio, SQL Server Integration Services (SSIS), SQL Server
Analysis Services (SSAS), SQL Server Reporting Services (SSRS), SQL Server 2014/2012/2008 R2, T-SQL, MSSQL, Power BI
|
