| | | 20,000+ Fresh Resumes Monthly | |
|
|
| |
Click here or scroll down to respond to this candidateCandidate's Name
Phone: PHONE NUMBER AVAILABLE Email: EMAIL AVAILABLE LinkedIn: LINKEDIN LINK AVAILABLESUMMARYExperienced Data Engineer with around 6 years of expertise in data modeling, data mining, data cleaning, and data visualization, skilled in developing and maintaining data pipelines using both AWS and Azure, leveraging big data tools for large-scale data processing and analysis.Proficient in batch processing and real-time streaming, ensuring efficient data handling and processing with Apache Flink and Spark-Streaming technologies.Utilized version control systems like GIT and Bitbucket for source code management and collaboration.Leveraged DAGs and Yarn for job scheduling and resource management, ensuring efficient task execution and resource allocation.Proficient with diverse tools and technologies, including Unix, Shell Scripting, Control-M, Sqoop, Avro, Parquet, ORC, and JSON.Collaborated with cross-functional teams in an Agile environment, contributing to sprint planning, daily stand-ups, and retrospectives.Mentored junior developers, providing guidance and support to help them grow their skills and succeed in their roles.Excellent communication skills, a quick learner of new technologies, and open to relocation across the USA.Team player with the ability to work effectively and efficiently with others, being a self-starter with a strong sense of self-motivation.SKILLSCloud Platforms:AWS: EC2, EMR, Glue, S3, Athena, Redshift, DynamoDB, Kinesis, Lambda, CloudFormation, EKS, CloudWatch, SNS, SQS, Amazon RDS, AWS AthenaAzure: Azure Data Factory, Azure Synapse Analytics, Azure Databricks, Azure DevOpsFrameworks and Processing: Hadoop, Spark, MapReduce, Pig, Hive, PySpark, Apache Flink, Spark-Streaming, Apache NiFi, Talend, Airflow, Oozie, DBTProgramming Languages: Java, Scala, Python, Unix Shell Scripting, R, C, C++, C#DevOps and CI/CD: Jenkins, Ansible, Terraform, Azure DevOps, Git, Maven, ServiceNow, Chef, Puppet, IAM Policies, Access Controls, Roles, AWS CloudWatch, BitbucketETL and Streaming: Apache Spark, Talend, Airflow, Oozie, Change Data Capture (CDC), KafkaDatabases and Storage: RDBMS, HDFS, HBase, Impala, Cloudera, Hortonworks, NoSQL, MongoDB, Cassandra, Snowflake, Oracle, MySQL, PostgreSQL, Teradata, Cosmos DBData Analysis: Power BI, Tableau, Databricks, AWS QuickSightOther Skills/Tools: Jira, ServiceNow, Code reviews, Collaborative development, Leadership, Proactive learning, LLMsCERTIFICATIONSAWS Certified Data Engineer AssociateAzure Data Engineer AssociateAWS Certified Cloud PractitionerEXPERIENCEGTS & Data EngineerNorthern Arizona University Mar 2023 Present, Flagstaff, AZDeveloped and maintained data pipelines for student and faculty outreach, leveraging Azure Data Factory to automate data ingestion and transformation from various sources into Azure Data Lake. Managed and resolved ServiceNow tickets, ensuring timely support and issue resolution.Administered Azure data services and ensured data security, configuring access controls, roles, and permission sets to meet security requirements.Demonstrated experience in DevOps practices using Azure DevOps, implementing Continuous Integration/Continuous Deployment (CI/CD) pipelines for data workflows.Utilized Azure Managed Services such as Azure Synapse Analytics and Azure Databricks for data processing and analysis, ensuring reliable and scalable data solutions.Tutored graduate students in Python programming, Data Structures, and Large-scale data structures, ensuring in-depth understanding and application of these key subjects.Learned and tutored fuzzy matching, heuristic, and exhaustive search algorithms such as Needleman-Wunsch, Smith-Waterman, and BLAST in large-scale data structures.Cloud Data EngineerTechfosys Software Solutions Jul 2021 Dec 2022, Chennai, IndiaMigrated data from legacy on-premises systems to AWS cloud, ensuring minimal downtime and data integrity.Utilized a broad array of AWS services, including EC2, EMR, Glue, Glue Catalog, S3, Athena, Redshift, DynamoDB, Kinesis, Lambda, CloudFormation, EKS, CloudWatch, SNS, and SQS, to support robust data migration processes.Implemented data warehousing solutions with Snowflake and Snowpark, enabling efficient storage and querying of large datasets.Deployed Terraform for infrastructure as code (IaC) to automate the provisioning and management of cloud resources.Conducted data modeling to design scalable and efficient data structures, supporting both OLAP and OLTP systems.Built and optimized data warehouses using Redshift and Snowflake, enhancing data accessibility and performance.Implemented data mining techniques to uncover insights and patterns, driving informed business decisions.Developed and maintained data warehouses, ensuring data consistency, reliability, and accessibility.Employed Hadoop, Hive, HBase, and HDFS for large-scale data storage and processing, ensuring high availability and fault tolerance.Utilized Impala and Cloudera/Hortonworks distributions to enhance big data processing capabilities.Managed clusters using Docker and Kubernetes, ensuring efficient resource utilization and scalability.Developed interactive dashboards and reports using Power BI and Tableau, providing actionable insights to stakeholders.Utilized DBT (Data Build Tool) for data transformations and modeling, ensuring data accuracy and consistency. Conducted data cleaning and preprocessing using Python, Scala, and PySpark, improving data quality for analysis.Automated infrastructure development and deployment using AWS CloudFormation, Terraform, Ansible, Tekton, and Jenkins.Implemented CI/CD pipelines for continuous integration and delivery, streamlining development and deployment processes.Conducted unit testing and performance optimization to ensure the reliability and efficiency of data pipelines.Enforced IAM policies and security best practices to safeguard sensitive data and ensure compliance with regulatory requirements.Implemented monitoring and logging solutions using AWS CloudWatch to track system performance and identify potential issues.Designed and developed ETL and ELT pipelines using Apache Spark, Talend, and AWS Glue, facilitating the seamless extraction, transformation, and loading of data from diverse sources.Managed data integration workflows with Airflow and Oozie, ensuring timely and reliable data processing.Leveraged Kafka for real-time data streaming and integration, enabling low-latency data flows across the organization.Utilized Databricks and Databricks UI/notebooks for interactive data analysis and processing, implementing the medallion architecture for data organization.Implemented Change Data Capture (CDC) techniques to maintain data accuracy and consistency across systems.Leveraged PyTorch and machine learning techniques to develop predictive models and enhance data-driven decision-making.Data EngineerTata Consultancy Services Nov 2018 Jun 2021, Hyderabad, IndiaOptimized big data applications by tuning performance, adjusting system parameters, and refining code.Developed and maintained Unix shell scripts to automate data processing tasks and system monitoring.Implemented unit tests using JUnit and ScalaTest, using Git for version control and Maven for project build management.Develop and optimize code in Java, Scala, and Python for data processing, leveraging Hadoop, Spark, MapReduce, Pig, Hive, and PySpark for large-scale analytics.Implement real-time data processing solutions with Kafka, Flink, Spark-Streaming, and Apache NiFi, and design microservices and RESTful APIs for data integration.Utilized Jenkins for continuous integration, automated deployment, and managing projects using Jira.Designed and managed DAGs for complex data pipeline orchestration and used NoSQL databases like MongoDB and Cassandra for data storage.Developed data extraction, transformation, and replication processes using Apache NiFi and pandas for data manipulation and analysis.Implemented infrastructure automation with Chef and Puppet and developed ETL processes to integrate data from various sources.Implemented front-end UI components using HTML5, CSS3, Ajax, JavaScript, Vue.Js, Bootstrap, jQuery, NodeJS, and ReactJS.Set up a Continuous Delivery (CI/CD) pipeline using Docker, Jenkins, GitHub, and AWS AMIs.Automated infrastructure development on AWS (EC2, DynamoDB, Lambda, EBS, RDS, ELB, EIP, etc.) using AWS CloudFormation.Utilized SoapUI to test RESTful and SOAP web services, ensuring robust and reliable API integrations.PROJECTSInnovative Applications in Hardware Security Using ChatGPT 3.5 for Bug Remediation and Vulnerability InsertionData Visualization Project with AWS QuickSight and S3 for Dynamic DashboardsPortfolio Website Development and Hosting Using AWS S3 and Route 53AWS-based ETL Solution with AWS Glue, Lambda, S3, and IAM for Data Transformation and Workflow Automation.EDUCATIONMasters degree in Computer ScienceNorthern Arizona University, Flagstaff, AZ May 2024, 4.0 GPABachelors degree in Electrical and Electronics EngineeringGayatri Vidya Parishad College of Engineering, Visakhapatnam, India May 2018 |
