| | | 20,000+ Fresh Resumes Monthly | |
|
|
| |
Click here or scroll down to respond to this candidateCandidate's Name
Sr. Data EngineerEmail id:EMAIL AVAILABLEContact no: PHONE NUMBER AVAILABLEProfessional Summary6+ years of professional experience in information technology as Data Engineer with an expert hand in the areas of Database Development, ETL Development, Data modelling, Report Development and Big Data Technologies.Experience in Data Integration and Data Warehousing using various ETL tools InformaticaPowerCenter, AWS Glue, SQL Server Integration Services (SSIS), Talend.Experience in Designing Business Intelligence Solutions with Microsoft SQL Server and using MS SQL Server Integration Services (SSIS), MS SQL Server Reporting Services (SSRS) and SQL Server Analysis Services (SSAS).Extensively used Informatica PowerCenter, Informatica Data Quality (IDQ) as ETL tool for extracting, transforming, loading and cleansing data from various source data inputs to various targets, in batch and real time.Experience working with Amazon Web Services (AWS) cloud and its services like Snowflake, EC2, S3, RDS, EMR, VPC, IAM, Elastic Load Balancing, Lambda, RedShift, Elastic Cache, Auto Scaling, Cloud Front, Cloud Watch, Data Pipeline, DMS, Aurora, ETL and other AWS Services.Strong expertises in Relational Data Base systems like Oracle, MS SQL Server, TeraData, MS Access, DB2 design and database development using SQL, PL/SQL, SQL PLUS, TOAD, SQL - LOADER. Highly proficient in writing, testing and implementation of triggers, stored procedures, functions, packages, Cursors using PL/SQL.Hands on Experience with AWS Snowflake cloud data warehouse and AWS S3 bucket for integrating data from multiple source system which include loading nested JSON formatted data into Snowflake table.Extensive experience in integration of Informatica Data Quality (IDQ) with InformaticaPowerCenter.Extensive experience in Data Mining solutions to various business problems and generating data visualizations using Tableau, PowerBI, Alteryx.Well knowledge and experience in Cloudera ecosystem such as HDFS, Hive, SQOOP, HBASE, Kafka, Data pipeline, Data analysis and processing with Hive SQL, IMPALA, SPARK, SPARK SQL.Worked with different scheduling tools like Talend Administrator Console(TAC), UC4/Atomic, Tidal, Control M, Autosys, CRON TAB and TWS (Tivoli Workload Scheduler).Experienced in design, development, Unit testing, integration, debugging and implementation and production support, client interaction and understanding business application, business data flow and data relations.Using Flume, Kafka and Spark streaming to ingest real time or near real time data to HDFS.Analysed data and provided insights with Python Pandas.Worked on AWS Data Pipeline to configure data loads from S3 into Redshift.Worked on Data Migration from Teradata to AWS Snowflake Environment using Python and BI tools like Alteryx.Developed Python scripts to parse the Flat Files, CSV, XML, JSON files and extract the data from various sources and load the data into data warehouse.Developed Automated scripts to do the migration using Unix shell scripting, Python, Oracle/TD SQL, TD Macros and Procedures.Good Knowledge on No SQL database like HBase, Cassandra.Expert-level mastery in designing and developing complex mappings to extract data from diverse sources including flat files, RDBMS tables, legacy system files, XML files, Applications, COBOL Sources & Teradata.Worked on JIRA for defect/issues logging & tracking and documented all my work using CONFLUENCE.Experience with ETL workflow Management tools like Apache Airflow and have significant experience in writing the python scripts to implement the workflow.Experience in identifying Bottlenecks in ETL Processes and Performance tuning of the production applications using Database Tuning, Partitioning, Index Usage, Aggregate Tables, Session partitioning, Load strategies, commit intervals and transformation tuning.TECHNICAL SKILLS :ETLInformatica Power Center 10.x/9.6/9.1, AWS Glue, Talend 5.6, SQL Server Integration Services (SSIS)Databases & Tools:MS SQL Server 2014/2012/2008, Teradata 15/14, Oracle 11g10g, SQL Assistant, Erwin 8/9, ER StudioCloud EnvironmentAWS Snowflake, AWS RDS, AWS Aurora, Redshift, EC2, EMR, S3, Lambda, Glue, Data Pipeline, Athena, Data Migration Services, SQS,Reporting ToolsTableau, PowerBIBig Data EcosystemHDFS, Map Reduce, Hive/Impala, Pig, Sqoop, Hbase, Spark, Scala, KafkaProgramming languagesUnix Shell Scripting, SQL, PL/SQL, Perl, Python, T-SQLData Warehousing & BIStar Schema, Snowflake schema, Facts and Dimensions tables, SAS, SSIS, and SplunkProfessional ExperienceClient: Wells Fargo, San Francisco, CA April 2022 to PresentRole: Sr. Data EngineerResponsibilities:Experience in building and architecting multiple Data pipelines, end to end ETL and ELT process for Data ingestion and transformation.Perform Informatica Cloud Services, Informatica Power Center Administration ETL strategies and ETL Informatica mapping. Setting up of Secure Agent and connect different applications and its Data Connectors for processing the different kinds of data including unstructured (logs, click streams, Shares, likes, topics etc..), semi structured (XML, JSON) and structured like RDBMS.Worked extensively with AWS services like EC2, S3, VPC, ELB, Auto Scaling Groups, Route 53, IAM, CloudTrail, CloudWatch, CloudFormation, CloudFront, SNS, and RDS.Building a Scala and spark based configurable framework to connect common Data sources like MYSQL, Oracle, Postgres, SQL Server, Salesforce, Big query and load it in Big query.Extensive Knowledge and hands-on experience implementing PaaS, IaaS, SaaS style delivery models inside the Enterprise (Data centre) and in Public Clouds using like AWS, Google Cloud, and Kubernetes etc.Worked on documentation of al worked Extract. Transform and Load, Designed, developed and validated and deployed the Talend ETL processes for the Data Warehouse team using PIG, Hive.Applied required transformation using AWS Glue and loaded data back to Redshift and S3.Extensively worked on making REST API (application program interface) calls to get the data as JSON response and parse it.Experience in analyzing and writing SQL queries to extract the data in Json format through Rest API calls with API Keys, ADMIN Keys and Query Keys and load the data into Data warehouse.Extensively worked on Informatica tools like source analyzer, mapping designer, workflow manager, workflow monitor, Mapplets, Worklets and repository manager.Building data pipeline ETLs for data movement to S3, then to Redshift.Worked on Data Extraction, aggregations and consolidation of Adobe data within AWS Glue using PySpark.Developed SSIS packages to Extract, Transform and Load ETL data into the SQL Server database from the legacy mainframe data sources.Worked on Postman using HTTP requests to GET the data from RESTful API and validate the API calls.Hands-on experience with Informatica power center and power exchange in integrating with different applications and relational databasesPrepared dashboards using Tableau for summarizing Configuration, Quotes, Orders and other e-commerce data.Created Informatica workflows and IDQ mappings for - Batch and Real Time.Provided Best Practice document for Docker, Jenkins, Puppet and GIT.Expertise in implementing DevOps culture through CI/CD tools like Repos, Code Deploy, Code Pipeline, GitHub.Backing up AWSPostgrestoS3on daily job run onEMRusing Data Frames.Developed server-based web traffic using RESTful API's statistical analysis tool using Flask, Pandas.Analyse various type of raw file like Json, Csv, Xml with Python using Pandas, Numpy etc.Environment: Informatica Power Center 10.x/9.x, IDQ, AWS Redshift, Snowflake, S3, Postgres, MS SQL Server, Big query, Salesforce Sql, Python, Postman, Tableau, Unix Shell Scripting, EMR, GitHub.Client: Cummins Columbus, Indiana Nov 2021 to Mar 2022Role: Data EngineerResponsibilities:Involved in full Software Development Life Cycle (SDLC) - Business Requirements Analysis, preparation of Technical Design documents, Data Analysis, Logical and Physical database design, Coding, Testing, Implementing, and deploying to business users.Design Setup maintain Administrator the Azure SQL Database, Azure Analysis Service, Azure SQL Data warehouse, Azure Data Factory, Azure SQL Data warehouse.Design and implement database solutions in Azure SQL Data Warehouse, Azure SQL .Changing the existing Data Models using Erwin for Enhancements to the existing Data warehouse projects.Used Talend connectors integrated to Redshift - BI Development for multiple technical projects running in parallel.Using g-cloud function with Python to load Data in to Big query for on arrival csv files in GCS bucket.Created iterative macro in Alteryx to send Json request and download Json response from webservice and analyze the response data.Supported various business teams with Data Mining and Reporting by writing complex SQLs using Basic and Advanced SQL including OLAP functions like Ranking, partitioning and windowing functions, Etc.Experience working with Windows Hyper-V Server, Azure, Windows Clustering including.Expertise in writing scripts for Data Extraction, Transformation and Loading of data from legacy systems to target data warehouse using BTEQ, FastLoad, MultiLoad, and Tpump.Tuning SQL queries using Explain analyzing the data distribution among AMPs and index usage, collect statistics, definition of indexes, revision of correlated sub queries, usage of Hash functions, etcExtensively used spark SQL and Data frames API in building spark applications.Written complex SQLs using joins, sub queries and correlated sub queries. Expertise in SQL Queries for cross verification of data.Worked Azure SQL Database Environment.Extensively worked on performance tuning of Informatica and IDQ mappings.Experience on data profiling & various data quality rules development using Informatica Data Quality (IDQ).Create new UNIX scripts to automate and to handle different file processing, editing and execution sequences with shell scripting by using basic Unix commands and awk,sed editing languages.Experience in cloud versioning technologies like GitHub.Integrate Collibra with Data Lake using Collibra connect API.Create firewall rules to access Google Dataproc from other machines.Write Scala program for spark transformation in Dataproc.Providing technical support and guidance to the offshore team to address complex business problems.Environment: Informatica Power Center 9.5,Talend,, Azure, PostgreSQL Server, Python, Oracle, Teradata, CRON, Unix Shel Scripting, SQL, Erwin.Role: Data EngineerClient: Ditech,India Feb2018 to Aug 2021Responsibilities:Involved in gathering business requirements, logical modeling, physical database design, data sourcing and data transformation, data loading, SQL, and performance tuning.Used SSIS to populate data from various data sources, creating packages for different data loading operations for applications.Performed data analytics on DataLake using pyspark on databricks platformDesigned and documented the entire Architecture of Power BI POC.Implementation and delivery of MSBI platform solutions to develop and deploy ETL, analytical, reporting and scorecard / dashboards on SQL Server using SSIS, SSRS.Extensively worked with SSIS tool suite, designed and created mapping using various SSIS transformations like OLEDB command, Conditional Split, Lookup, Aggregator, Multicast and Derived Column.Scheduled and executed SSIS Packages using SQL Server Agent and Development of automated daily, weekly and monthly system maintenance tasks such as database backup, Database Integrity verification, indexing and statistics updates.Developed and maintained data pipelines and ETL processes, ensuring efficient data extraction, transformation, and loading.Worked extensively on SQL, PL/SQL, and UNIX shell scripting.Expertise in creating PL/ SQL Procedures, Functions, Triggers and cursors.Loading data in No SQL database (HBase, Cassandra)Expert level knowledge of complex SQL using Teradata functions, macros and stored procedures.Developing under scrum methodology and in a CI/CD environment using Jenkin.Deploy EC2 instances for oracle database.Utilized Power Query in Power BI to Pivot and Un-pivot the data model for data cleansing.Design & implement migration strategies for traditional systems on Azure (Lift and shift/Azure Migrate, other third-party tools.Used various sources to pull data into Power BI such as SQL Server, Excel, Oracle, SQL Azure etc.Propose architectures considering cost/spend in Azure and develop recommendations to right-size data infrastructure.Design Setup maintain Administrator the Azure SQL Database, Azure Analysis Service, Azure SQL Data warehouse, Azure Data Factory, Azure SQL Data warehouse.Worked Azure SQL Database Environment.Experience working with Windows Hyper-V Server, Azure, Windows Clustering including.Environment: MS SQL Server 2016, ETL, SSIS, SSRS, SSMS, Cassandra, Oracle 12c, Oracle Enterprise Linux, Teradata, Databricks, Jenkins, PowerBI, Autosys, Unix Shell Scripting, Azure.Role: Data AnalystClient: Thomson Reuters, India Jun 2017 to Jan 2018Responsibilities:Performed Data Analysis, Data Migration, Data Cleansing, Transformation, Integration, Data Import, and Data Export.Worked on Design, Development and Documentation of the ETL strategy to populate the data from the various source systems using Talend ETL tool into Data Warehouse.Devised PL/SQL Stored Procedures, Functions, Triggers, Views and packages. Made use of Indexing, Aggregation and Materialized views to optimize query performance.Created Tableau dashboards/reports for data visualization, Reporting and Analysis and presented it to Business.Design and develop spark job with Scala to implement end to end data pipeline for batch processingCreated Data Connections, Published on Tableau Server for usage with Operational or Monitoring Dashboards.Knowledge in Tableau Administration Tool for Configuration, adding users, managing licenses and data connections, scheduling tasks, embedding views by integrating with other platforms.Experience in Converting existing AWS Infrastructure to Server less architecture (AWS Lambda, Kinesis), deploying via Terraform and AWS Cloud Formation templates.Developed data ingestion models to data into various layers in S3, Redshift, Snowflake, using AWS Glue, AWS Step functions, AWS Lambda.Primarily Responsible for converting Manual Report system to fully automated CI/CD Data Pipeline that ingest data from different Marketing platform to AWS S3 data lake.Utilized AWS services with focus on big data analytics, enterprise data warehouse and business intelligence solutions to ensure optimal architecture, scalability, flexibilityDesigned AWS architecture, Cloud migration, AWS EMR, DynamoDB, Redshift and event processing using lambda function .Used AWS system manager to automate operational tasks across AWS resources.Extracted Mega Data from AWS using SQL Queries to create reports.Environment: Hadoop Ecosystem (HDFS), Talend, SQL, Tableau, Hive, Sqoop, Kafka, Impala, Spark, Unix Shell Scripting.Education: Bachelors in Computer Science from Jawaharlal Nehru Technological University, 2018. |
