| | | 20,000+ Fresh Resumes Monthly | |
|
|
| |
Click here or scroll down to respond to this candidate Candidate's Name
Data EngineerEmail: EMAIL AVAILABLE Phone: PHONE NUMBER AVAILABLE
PROFESSIONAL SUMMARY:Highly skilled and results-driven Data Engineer with over 3+ years of experience in designing and implementing complex data solutions. Proficient in SQL query writing, PostgreSQL, AWS Redshift, AWS cloud services, and Azure cloud services, with a proven track record of expanding data platform capabilities to address emerging challenges. Skilled in Python programming, Linux bash scripting, Spark, Hadoop, Hive, and various cloud platforms. Adept at collaborating with cross-functional teams to ensure the delivery of scalable and efficient data solutions. Possesses a strong understanding of data modeling principles and agile methodologies, combined with excellent analytical and troubleshooting abilities. Additionally, well-versed in machine learning algorithms and predictive modeling, with experience in implementing models using Python, scikit-learn, and TensorFlow, along with exposure to Big Data ecosystems. Proficient in Power BI and Tableau for comprehensive data visualization.TECHNICAL SKILLS:BigData/HadoopMapReduce, Spark, Kafka, PySpark, Pig, Hive, HBase, Flume, Oozie, ZookeeperLanguagesPython (NumPy, SciPy, Pandas, Genism, Keras), Java, Java Script, Shell ScriptingNOSQL DBCassandra, HBase, MongoDB, MariaDBDev ToolsMicrosoft SQL Studio, IntelliJ, Azure Databricks, Eclipse, Net Beans.CloudAWS
Build ToolsJenkins, Toad, SQL Loader, PostgreSQL, Talend, Maven, ANT, RTC, RSA, Oozie, Hue, SOAP UIReporting ToolsMS Office (Word/Excel/Power Point/ Visio/Outlook), Crystal reports XI, SSRS, Cognos.DatabasesMS SQL Server, MySQL, Oracle, DB2, Teradata, NetezzaOperating SysWindows, UNIX, LINUXWORK EXPERIENCE:Client: Broadridge Financial Solutions / Hyderabad, India Sep 2022 PresentRole: Data EngineerResponsibilities: Implemented Hadoop framework to capture user navigation data, facilitating UI validation and providing analytic feedback to the UI team. Loaded data into the cluster from dynamically generated files using Flume and relational database management systems using AWS Glue. Tuned Spark Applications for optimal performance, adjusting batch interval time, parallelism levels, and memory allocation. Built real-time streaming pipeline with AWS Kinesis, AWS Glue, and Amazon Redshift for continuous data processing. Developed logical and physical data flow models for Informatica ETL applications to streamline data transformation processes. Created custom Docker container images and managed image tagging and deployment processes. Wrote Hive queries for data analysis to meet business requirements and extracted insights. Analyzed and optimized SQL query performance in databases to enhance data processing efficiency. Loaded and transformed structured, semi-structured, and unstructured data from Amazon DynamoDB using AWS Glue. Coded, tested, and documented new or modified data systems, contributing to the development of a robust and scalable data platform. Ensured consistency of data solutions by collaborating closely with developers, adhering to established standards and best practices. Expanded data platform capabilities to address evolving challenges, fostering innovation and efficiency within the organization. Played a key role in creating and maintaining comprehensive data catalogs, facilitating efficient data management and accessibility. Designed logical models and implemented physical databases to support business needs, optimizing data integrity and performance. Reviewed and approved database changes according to design standards, ensuring data consistency and reliability. Analyzed and resolved technical and application problems, providing timely support, and troubleshooting for production issues and code deployment activities. Leveraged AWS services to develop, host, and maintain data extraction, transformation, and loading functions, optimizing data workflows for enhanced efficiency. Utilized AWS Glue, T-SQL, and Spark SQL for data ingestion and processing in AWS Databricks. Created and managed data pipelines in AWS Data Pipeline to extract, transform, and load data from various sources. Developed basic Tableau visualizations to showcase sales trends, contributing to team presentations and client meetings. Collaborated with team members to document data analysis processes and create user guides for Tableau dashboards. Presented data findings using Tableau, Power BI, and Google Data Studio, ensuring clear and understandable visualization of insights. Automated data extraction and transformation processes using Tableau Prep and maintained up-to-date dashboards and reports using Power BI Gateways. Developed reports with time intelligence features such as Year to Date (YTD) and Month to Date (MTD) analysis using Power BI. Created JSON scripts for deploying data pipelines in AWS Data Pipeline, incorporating SQL activity. Utilized ELK cluster for organizing error codes and log information, setting up alerts in Kibana dashboard for continuous monitoring of Spark applications and triggering emails for any job failures.Client: Mouri Tech / Hyderabad, India Oct 2020 Aug 2022Role: Data EngineerResponsibilities: Utilized AWS Glue Catalog with crawler to extract and catalog data from S3, facilitating seamless data accessibility and analysis through AWS Athena. Implemented process enhancements in data workflows using Alteryx processing engine and SQL, resulting in improved data processing efficiency and accuracy. Developed Terraform scripts to automate provisioning of AWS infrastructure components such as ELB, CloudFront distribution, RDS, EC2 instances, and S3 buckets, optimizing deployment processes. Integrated Apache Airflow with AWS services to orchestrate and monitor Machine Learning workflows on SageMaker, ensuring smooth execution and monitoring of ML tasks. Collaborated closely with the Data Science team to build and deploy machine learning models on Spark EMR cluster, meeting specific business requirements and enhancing data-driven decision-making processes. Designed and developed MapReduce/Spark Python modules for Machine Learning applications in Hadoop on AWS, facilitating efficient data processing and analysis. Led the migration of PostgreSQL and MySQL databases to AWS Aurora, ensuring seamless migration and optimization of database performance. Demonstrated expertise in designing and implementing ETL workflows, leveraging AWS Glue, AWS Data Pipeline, and AWS Step Functions to streamline data extraction, transformation, and loading processes. Developed Spark applications using PySpark and Spark SQL for data extraction, transformation, and aggregation tasks across diverse file formats, enhancing data processing capabilities. Automated data processing tasks using AWS Lambda functions written in Python, improving efficiency in handling nested JSON files and other data formats. Designed and implemented a Snowflake warehouse strategy, migrating terabytes of data from S3 into Snowflake using PUT scripts, ensuring efficient and scalable data management. Leveraged coding and scripting pipelines, along with AWS services like Athena Data Catalog, to transform data into valuable assets, enabling efficient data exploration and analysis. Utilized Looker to generate customized dashboards, reports, and visualizations, empowering stakeholders with actionable insights and facilitating data-driven decision-making. Employed Elastic search to perform efficient searching, analysis, and visualization of large volumes of structured and unstructured data, catering to diverse analytical use cases. Developed PySpark jobs in AWS Glue to merge data from multiple sources, populating the AWS Glue data catalog with metadata table definitions, ensuring accurate cataloging and management of data. Created and executed ETL processes in AWS Glue to migrate campaign data from various sources like S3, ORC, Parquet, and Text Files into AWS Redshift, ensuring seamless data integration and storage.
Worked in an Agile Development environment.Education Details:Bachelor s in computer science - Jawaharlal Nehru Technological University, Hyderabad (Nov, 2020) |
