| | | 20,000+ Fresh Resumes Monthly | |
|
|
| |
Click here or scroll down to respond to this candidate?Harsha GowthamEMAIL AVAILABLE PHONE NUMBER AVAILABLEPROFESSIONAL SUMMARY? Over 9+ years of IT experience as a Developer, Designer & quality Tester with cross-platform integration experience using Hadoop development and Admin? Strong Experience in data engineering and building ETL pipelines on batch and streaming data using Pyspark, SparkSQL, RDD? Designed and implemented a scalable NoSQL database using Apache HBase for a high-volume, low-latency application, enabling the system to store and process large volumes of structured and unstructured data efficiently? Good working exposure to Cloud technologies of Azure - Virtual Machines, Data Lake Storage, Azure Functions, Azure Data Factory? Strong experience in Big Data technologies including Hadoop, Spark and Azure HDInsight? Proficient in performing EDA (Exploratory data analysis), Root Cause Analysis, and Impact Analysis on large volumes of datasets? Solid understanding of RDBMS database concepts including performance tuning and Query optimization? Experienced in querying Snowflake, Azure Synapse, SQL Server, and Azure Cosmos DB for OLAP and OLTP? Leveraged Apache Pig and Hive to build an ETL pipeline for a data warehousing project, allowing the team to extract, transform, and load data from multiple sources into a centralized data repository? Experience building and optimizing big data pipelines, architectures and data sets on building Data Lakes? Experience with stream-processing systems using Kafka with zookeeperTECHNICAL SKILLS? Programming Languages: Python, Java, C, SQL, NoSQL, Javascript, MySQL? Scripting Languages: Linux, Bash scripting, Shell scripting? Machine Learning Libraries: Numpy, Pandas, Keras, Tensorflow, SciPy, Matplotlib? Tools: Snowflake, Apache Hive, Redshift, Informatica, Azure Data Factory, Azure Databricks, Azure Data Lake, Azure Functions, Azure Synapse, Azure Monitor, Jira, SSIS, SSRS, SSAS, SSMS? Databases: MySQL, PostgreSQL, MongoDB, Snowflake, Cassandra, Oracle, Azure Cosmos DB? Cloud Technologies:Azure (Virtual Machines, Data Lake Storage, Azure Functions, Azure Data Factory, Azure Databricks, Azure Synapse, Azure Event Hubs, Azure Stream Analytics)? Reporting tools: Tableau, Power BI, LookerEXPERIENCEUS Bank, Minneapolis, MinnesotaSr Data Engineer January 2022 - Present? Developed and implemented ETL workflows using Azure Data Factory with Azure Databricks, handling AVRO and Parquet file formats, to create efficient denormalized tables for data processing and storage? Led multiple projects leveraging dBT Labs for data engineering tasks, demonstrating proficiency in data manipulation, transformation, and analysis.? Conducted performance tuning and optimization of dbt models in Snowflake, improving query performance and report generation? Automated event-driven data processing tasks within data pipelines using Azure Functions, optimizing efficiency and streamlining data workflows? Improved data integrity and quality through the execution of data checks in Spark with Spark SQL and Linux scripts and scheduled ETL workflows with Airflow-DAG? Transformed data from multiple sources including APIs, databases using Azure Data Factory and Azure Databricks into a standardized format for OLAP? Designed event-driven architectures using Azure Event Hubs and Azure Service Bus for real-time analytics? Utilized Azure HDInsight to perform large-scale data processing including running PySpark jobs? Optimized data pipeline performance and ensured high availability using Azure Monitor, Azure Key Vault, and Azure Security Center, enabling efficient data processing and real-time monitoring? Experienced in leveraging Kafka Streams and Kafka Connect for real-time stream processing and data integration tasks? Designed external tables with partitioning and bucketing techniques using a shared meta-store in MySQL to increase query performance? Implemented Docker containerization best practices, including creating lightweight, efficient Docker images and optimizing container orchestration for cost-effectiveness? Integrated code repositories, version control systems, and issue tracking tools into CI/CD pipelines to ensure seamless collaboration and traceability across the development lifecycle? Proficient in leveraging Power BI tools and Tableau to create interactive dashboards and visualizations for data-driven decision makingBCBS, Chicago, ILETL Developer August 2021 - December 2021? Designed and implemented the data pipeline architecture using Azure services such as Azure Blob Storage, Azure Functions, allowing for real-time processing of high-frequency data.? Optimized Snowflake configurations and performance tuning parameters to enhance query execution speed and overall system efficiency? Utilized Python extensively to develop custom scripts and automation tools for data processing, ensuring efficient and scalable solutions within the dBT environment? Built an ETL pipeline to extract, transform, and load data from multiple sources (DynamoDB, RDS, and APIs) into a data lake Azure Blob Storage using Lambda triggers? Implemented integrated OLTP and OLAP systems for streamlined data processing and advanced analytics? Leveraged Azure Data Factory for data transformation and glue job orchestration, creating efficient data processing workflows using Python and Spark? Developed and maintained dbt models in Snowflake, ensuring accurate and consistent data structures and calculations across the data warehouse? Developed ETL processes to extract, transform, and load trading data from multiple sources such as stock exchanges, market data providers, and news feeds, using tools such as Apache Kafka and Spark Streaming? Designed and implemented data models and schemas in Snowflake to facilitate efficient querying and data organization for further analysis? Built a data model for trading data using NoSQL databases such as MongoDB and Cassandra, allowing for efficient storage and retrieval of real-time data? Implemented data quality checks and monitoring using Azure Monitor and Azure Log Analytics, ensuring data accuracy and reliability for trading decisions.? Proficient in using industry-leading CI/CD tools such as Jenkins to automate build, test, and deployment workflows? Built interactive dashboards using Tableau, presenting insightful visualizations to stakeholdersCummins, Columbus, IndianaBig Data Engineer April 2017 - July 2021? Implemented end-to-end ETL processes using SQL Server Integration Services (SSIS) to extract, transform, and load data from flat files. Proficient in MSSQL, SQL SSIS, and SQL SSAS for seamless data integration? Designed and implemented efficient data models and mappings for accurate data integration between source and target systems (STT)? Optimized data storage and memory utilization for Azure Synapse, enhancing performance and query response times? Exhibited expertise in Data Warehousing concepts, including Fact tables, Dimension tables, and Star/Snowflake Schema modeling, while designing data extraction and integration processes using Talend for data warehouse? Integrated Azure Functions with Azure Logic Apps to build serverless architectures and enable seamless invocation of serverless functions through API endpoints? Have experience in using Python with PySpark in building data pipelines and writing python scripts to automate pipelines? Developed Spark applications for performing data cleansing, event enrichment, data aggregation, de-normalization and data preparation needed for machine learning exercises? Conducted performance tuning and optimization of Airflow and Azure Synapse configurations, improving data processing speed and reducing resource consumption? Created user-defined functions (UDF) in Azure Synapse and PySpark? Integrated security and compliance checks into CI/CD pipelines, including static code analysis, vulnerability scanning, and dependency management, to ensure secure and reliable software releasesLycatech Services Pvt Ltd, chennaiData Engineer September 2015 - March 2017? Designed, developed, and implemented ETL workflows using Apache NiFi to process data from various sources to a centralized data lake? Conducted regular monitoring and maintenance activities, including capacity planning and resource optimization, to ensure the smooth operation of data warehouses? Developed data models using ERD and UML to capture business requirements and support database design? Designed and implemented an ETL pipeline to extract, transform, and load data from multiple sources (DynamoDB, RDS, and APIs) into a data lake on Azure Blob Storage using Logic Apps triggers? Built and maintained NoSQL databases (MongoDB, Cassandra) to support application development and data analytics? Optimized SQL queries and database indexes to improve query performance and reduce query execution time using Spark? Worked with a team of data engineers to design and implement a scalable data warehousing solution using Hadoop and Hive? Designed and developed ETL workflows using Talend to extract data from various sources, transform it, and load it into the data warehouse |
