Hi, I'm Raj!

Leading bank data modernization program which aims to publish the banking datasets in the on-prem data platforms (Teradata and Hadoop) to Snowflake on Microsoft Azure public cloud platform.

Role: IT System Owner

Platform: Data Lakehouse on Cloud

Technology Stack: Snowflake, Azure, TWS, Python Scripts, Collibra, IDMC

Highlights:

• Developed the Banking One Data Model which is implemented in Snowflake.
• Published ~100 banking datasets to Snowflake with data catalogues in Collibra.
• Published roadmap for the rest (~400) of the banking datasets.
• Automated the development of data ingestion framework config files.
• Automated the preparation of data catalogue YAML files from Excel data dictionaries.

Managed the banking data warehouse for a lending product with 10+ data sources and 25+ consumer groups for a US based investment bank.

Role: IT System Owner

Platform: Data Warehouse

Technology Stack: Teradata, Informatica PowerCenter, Macros, Stored Procedures, TWS, Autosys, Shell Scripts

Highlights:

• 10+ data sources feeding loan, account and client data into the data warehouse.
• 25+ consumer groups receive data.
• Supports multiple lending and deposit products.
• Managed a team of around 10 data engineers.

Designed the data lake storage layer and data ingestion framework and led delivery of the data pipelines for a US based investment bank.

Role: Designer, Technology Delivery Manager

Platform: Data Lake

Technology Stack: Hive, PySpark, SparkSQL, Parquet, Sqoop, TWS, Shell Scripts

Highlights:

• Pilot for onboarding the first of the many lending products.
• Framework is used to load 500+ tables using 1000+ batch jobs.

Designed and led delivery of the data warehouse from scratch for a US based Insurance provider.

Role: Data Modeler, ETL Designer

Platform: Data Warehouse

Technology Stack: Oracle, Informatica PowerCenter, Autosys, Shell Scripts

Highlights:

• 5+ data sources feeding employee attendance, contact center management and customer feedback data into the data warehouse.
• Multiple departments receive reports and dashboards.
• Managed a team of around 5 data engineers.

Designed and led delivery of the data distribution layer using a data virtualization platform for a US based investment bank.

Role: Designer, Technology Delivery Manager

Platform: Data Virtualization

Technology Stack: Tibco Data Virtualization

Highlights:

• Data virtualization brings data from multiple sources together without ETL.
• Provides a semantic layer over the data sources.
• Abstracts the platform specific technical know-how from the business analysts and users.
• Able to publish consumer specific views of the data with user entitlements.

Designed and led delivery of data pipelines that feed data into regulatory reporting systems for a US based investment bank.

Role: Designer, Technology Delivery Manager

Platform: Data Warehouse, Data Virtualization

Technology Stack: Teradata, Macros, SQL, Tibco Data Virtualization

Highlights:

• Feeds data into multiple regulatory reports like CCAR FR-Y14M.
• Users able to make adjustments to the data to refresh the reports.

Designed and led delivery of an analytics application for a US based investment bank to determine the pricing of new loans based on existing relationships that the client has with the firm.

Role: Designer, Technology Delivery Manager

Platform: Data Lake, Data Warehouse

Technology Stack: Hive, HiveQL, PySpark, Teradata, Stored Procedures, SQL, TWS, Spring Boot, REST API, Business Objects

Highlights:

• Extracts assets, liabilities, party and household information from 50+ datasets.
• Calculates revenue, expenses, profit at account and household level for 5+ million households.

Designed and led delivery of an analytics application for a US based investment bank to determine the mortgage loans eligible for rate adjustment. This involves calculating the cash flows and internal rate of return for eligible products.

Role: Designer, Technology Delivery Manager

Platform: Data Lake

Technology Stack: Hive, HiveQL, PySpark, TWS, Autosys

Highlights:

• Calculates cash flows and internal rate of return for the current and eligible mortgage products.
• Determines if the loan is eligible for rate adjustment based on a set of business rules.

Designed and led delivery of data pipelines that feed data into the risk models for a US based investment bank.

Role: Designer, Technology Delivery Manager

Platform: Data Warehouse

Technology Stack: Teradata, Macros, SQL

Highlights:

• Feeds loan portfolio data into market risk and credit risk models.
• Captures model outputs and distirbutes to the data consumer groups.

Designed and led delivery of an automated process to reconcile the KYC data feeding into the compliance processes for a US based investment bank.

Role: Designer, Technology Delivery Manager

Platform: Data Warehouse

Technology Stack: Teradata Macros, Python

Highlights:

• Reconciles incoming feed files with data stored in the data warehouse across ~20 files/tables.
• Reusable framework allows adding files/tables in future.
• Automated regression test of outbound xml files.

Designed and developed the Data Lineage web application which is used for impact analysis, identifying sources and consumers of datasets and tracking the process lineage from source to target interfaces.

Role: Designer, Developer

Platform: Web Application

Technology Stack: Python, Flask, Angular, DOT, GraphViz, NetworkX

Highlights:

• Discover metadata by parsing the code repositories for datasets and process modules.
• Build a lineage graph which associates the source and target datasets, interfaces and processes.
• Serve the metadata and lineage graphs via REST API.
• UI to view the dataset inventory, sources and consumers. Interface to analyze the impacted objects by input dataset. Interface to view the lineage graph which includes predecessors and successors.

Designed and developed the Data Quality validation framework using the open source Great Expectations package.

Role: Designer, Developer

Platform: Application

Technology Stack: Python

Highlights:

• Configure data quality validation rules for inbound datasets.
• Rules are applied on the dataset using the open source great expectation package.
• Application can be invoked using CLI or REST API end points. This allows the app to be integrated into a larger data ingestion / distribution framework.
• I plan to create an UI to view/configure the data quality rules and dataset metadata.

Designed and developed the Data Quality Anomaly Detection ML application using the open source XGBoost package.

Role: Designer, Developer

Platform: Application

Technology Stack: Python, Gradient Boosting Decision Tree Classifier model, Packages (xgboost, matplotlib, pandas, shap)

Highlights:

• Detect anomalies in the current day snapshot data.
• Gradient Boosting Decision Tree Classifier model tries to classify the data records as current vs prior day data.
• Anomalies are detected when the model classifies any record as current day data.
• Model output is explained using SHAP values and plots.
• Application can be invoked using CLI or REST API end points. This allows the app to be integrated into a larger data ingestion / distribution framework.
• I plan to create an UI to view/configure the data quality rules and dataset metadata.

Developed a data profiling application which uses a Natural Language Processing (NLP) Named Entity Recognition (NER) model to classify the data as PII vs Non-PII.

Role: Designer, Developer

Platform: Application

Technology Stack: Python, Natural Language Processing (NLP) package spaCy, Named Entity Recognition (NER) model

Highlights:

• Named Entity Recognition (NER) model from the NLP package spaCy is used to recognize the entities in the input dataset.
• The entity labels output from the model are further translated using rules to determine the information type and data class (PII, Non-PII).
• Application can be invoked using CLI or REST API end points. This allows the app to be integrated into a larger data ingestion / distribution framework.

Developed a CLI app / micro service to create asset definition for the datasets that will be published to the data catalog. The catalog can be queried (e.g. for datasets, data elements) using the OpenAI GPT LLM model. The data dictionary, system and business glossaries are used for the RAG context.

Role: Designer, Developer

Platform: Application

Technology Stack: Python, FastAPI, GenAI (OpenAI GPT LLM with RAG), LangChain, LangGraph

Highlights:

• Catalog asset definitions can be formatted as needed by the enterprise data catalog platform (e.g. Collibra, Atlan).
• Users can discover the datasets and data elements using natural language queries. This is achieved through the GenAI LLM application which uses the OpenAI GPT LLM model.
• Application can be invoked using CLI or REST API end points. This allows the app to be integrated into a larger data ingestion / distribution framework.

Designed and developed a serverless ETL orchestration framework in the Microsoft Azure public cloud platform. The framework covers data ingestion, curation and delivery aspects while providing configurable data governance controls.

Role: Designer, POC Developer

Platform: Application

Technology Stack: Python, Azure Functions, Azure Datalake Storage, Azure Data Factory, Azure Databricks

Highlights:

• The ETL orchestration framework is built using stateful Azure Durable Functions.
• Ingestion workflow is triggered by the arrival of inbound files.
• Transformation and Extract/Delivery workflows can be triggered based on pre-defined predecessor workflows. A Monitor workflow watches for the dependencies and posts trigger events to Azure Event Grid.
• The workflows can be configured to invoke azure data factory pipelines, azure databricks notebooks along with data governance control functions (data quality, file integrity check, etc) as pre-ingestion and post-extraction tasks.

Designed and led delivery of data pipelines as part of the lending sub-servicer onboarding for a US based investment bank.

Role: Designer, Technology Delivery Manager

Platform: Data Warehouse

Technology Stack: Teradata, Informatica Powercenter, SQL, Macros, MultiLoad

Highlights:

• Loosely coupled with the data source unlike before.
• Transformations are pushed down to the database where they run much quicker.
• Concurrent execution of smaller, easy to manage pipelines resulted in 65% reduction in ingestion time.

Developed data pipelines to calculate the balance flows across multiple products for a US based investment bank. This was aggregated and presented to stakeholders at various levels in the firm via a dashboard.

Role: Designer, Developer

Platform: Data Warehouse

Technology Stack: Teradata, Tableau

Highlights:

• Account balances are aggregated across the branch hierarchy (branch -> complex -> division -> country)
• Row level security is implemented using role based entitlements in Teratata and Tableau.
• Account balance history is established through the life of the loan accounts across vendor migrations.

Designed and led delivery of the data pipelines to feed customer and transaction data into an Anti Money Laundering solution for a bank in the Nordic region.

Role: Designer, Developer

Platform: Data Warehouse

Technology Stack: Informatica PowerCenter, Teradata

Highlights:

• Created data pipelines to deliver the customer and transaction data feeds to NICE Actimize for AML validations.
• Received 5/5 score in customer satisfaction survey post the project delivery.
• Turned around the project which was plagued with delays into a success story within the revised timelines.

Designed and developed the Data Reconciliation Hub application for a US based investment bank to reconcile data from various data sources feeding into the data warehouse and data lake.

Role: Designer, POC Developer

Platform: Data Warehouse, Data Lake

Technology Stack: Python, Angular

Highlights:

• Reconcile inbound file with the DW/DL table.
• Reconcile 2 DW/DL tables for same data points received from 2 different source systems.
• UI allows users to define reconciliation rules and apply filters on the source data to match the populations of the datasets being reconciled.
• Users can subscribe to the reconciliation rules on the datasets that they are interested in. Users can view the summary/detail results in the UI. They get a notification with the summary results.

Developed this website to exhibit my professional work experience.

Platform: Azure Cloud

Technology Stack: HTML, CSS, Bootstrap, Azure Blob Storage, Azure DevOps, Azure Functions

Highlights:

• Static website is hosted in Azure Blob storage and served via Azure CDN.
• Azure DevOps pipeline is setup to publish the website on push to GitHub repository.
• Azure serverless function is setup to log the visits to the website.

A CLI app / micro service to reconcile the source data with the reconciliation control measures received from the source. Reconciliation controls (columns, aggregates) can be configured.

Role: Designer, Developer

Platform: Application

Technology Stack: Python, Pandas, FastAPI

Highlights:

• Calculate the reconciliation measures based on the rules defined in the metadata management service.
• Application can be invoked using CLI or REST API end points. This allows the app to be integrated into a larger data ingestion / distribution framework.

A CLI app / micro service to centrally store, manage and serve the metadata to the data management framework. This covers datasets, data quality rules, data reconciliation rules, schedules and holidays.

Role: Designer, Developer

Platform: Application

Technology Stack: Python, FastAPI

Highlights:

• Store and serve the common metadata (datasets, data quality rules, data recon rules, schedules and holidays) to all other services in the data management framework.
• Application can be invoked using CLI or REST API end points. This allows the app to be integrated into a larger data ingestion / distribution framework.

A CLI app / micro service to ingest delimited text files into data lake tables using PySpark API.

Role: Designer, Developer

Platform: Application

Technology Stack: Python, FastAPI, PySpark

Highlights:

• Load delimited text files into Spark tables using metadata.
• Ingestion workflow can be configured to load regular or partitioned tables. Full or incremental loads are possible.
• Ingestion workflow can be run for current cycle date or any back dated cycle (i.e. override).
• Ingestion workflow can be configured to run pre-ingestion tasks such as data quality checks and data profiling.
• Ingestion workflow can be configured to run post-ingestion tasks such as data reconciliation. These tasks are other framework services themselves.
• Application can be invoked using CLI or REST API end points. This allows the app to be integrated into a larger data processing / management framework.

A CLI app / micro service to extract data from the data lake tables using PySpark API.

Role: Designer, Developer

Platform: Application

Technology Stack: Python, FastAPI, PySpark

Highlights:

• Extract data from Spark tables and output to a delimited file for distribution using metadata.
• Distribution workflow can be configured to output data in various formats.
• Distribution workflow can be run for current cycle date or any back dated cycle (i.e. override).
• Distribution workflow can be configured to run post-ingestion tasks such as data quality checks and data profiling. These tasks are other framework services themselves.
• Application can be invoked using CLI or REST API end points. This allows the app to be integrated into a larger data processing / management framework.

A CLI app / micro service to derive the effective business dates based on the batch schedules and holidays (NYSE, Federal Bank holidays).

Role: Designer, Developer

Platform: Application

Technology Stack: Python, FastAPI

Highlights:

• Generates the application run calendar based on the schedules and holidays defined in the metadata service.
• Serves the effective business date for the given schedule id. This can be used by data ingestion, curation and distribution processes (e.g. resolve file name date patterns, date filter in SQL queries).
• Application can be invoked using CLI or REST API end points. This allows the app to be integrated into a larger data ingestion / distribution framework.

A CLI app / micro service to capture the lineage at the dataset level.

Role: Designer, Developer

Platform: Application

Technology Stack: Python, FastAPI

Highlights:

• Captures the data lineage relationships as the pipelines run to ingest or distribute data.
• Users can view the lineage relationships as a graph which shows the feeds, datasets and processes in the image.
• Application can be invoked using CLI or REST API end points. This allows the app to be integrated into a larger data ingestion / distribution framework.