Gibson Wasukira

PART 2 - ARCHITECTURE REQUIREMENTS GATHERING

In this second part of the series, we shall gather information that will drive the architectural design of the data platform. This is the genesis of our data platform. At the end of this phase, we should have a slight inclination of what kind of data pipeline we will build

Our primary role as data engineers is to build data pipelines that deliver data to end users

Before we start building our pipeline, we need to get answers to the following questions:

• Who are the data platform stakeholders? -- These could include Business Analysts, Dashboard Consumers, REST API subscriptions
• What data needs to get consumed?
• Which data helps business make decisions?
• What systems are used to capture this data?
• What data governance policies should be in place? o Who will have access to this data? – Security and Compliance
• More importantly what is the budget?
• Government compliance requirements, what are limitations on the geographical location of our data

Determine the Stakeholders

Let’s say our stakeholders are:

• Marketing; Marketing Director, Usage and Retention Analyst
• Sales; Sales Director, Sales Agents, Sales Analyst
• Finance; CFO, Investment Analysts
• Business Analysts
• Customers with Back-end developers as a proxy
• IT; CTO, Security and Compliance
• CRM – surprised ??? will discuss the reverse-ETL

What data needs to get consumed?

• CFO wants his financial statements visualized
• Sales agents want to know what issues are being reported by customers
• The Sales Director:
- • Needs to know how many prospects are in the leads pipeline, how many were converted into accounts over a certain period
- • Wants to track his Sales Agents performance
- • Wants daily visibility on how different products are performing
• Sales Analyst whose role is to build promotional products, has a rewards/loyalty program that requires real time data feeds to identify who qualifies for a reward and who doesn’t
• Marketing Director is so big on retention, he wants real time updates on usage/access patterns, for instance how long a customer spends looking at a certain page, and which page was clicked after, did that session convert into a sale?
• Business Analysts (BAs) need access to sales raw data, customer complaints and query resolution time
• Customer needs to view some sort of statement on demand through the mobile app
• CTO and Scrum master need to know bugs being reported, how long it took to fix them
• CTO also needss know the application traffic access patterns
• Data Scientist ?? His/Her List is exhaustive

What governance policies should be in place?

• Business Analysts (BAs) should only view sales and customer complaints datasets
• CFO, CTO, Marketing Director, Investment Analyst should be able to view all datasets
• Sales Agents should only view sales data

Remember to always follow the Principle of Least Privilege

Compliance

• Government Restrictions on the geographic location of the data, there are no restrictions for out hypothetical case

What is the budget?

• CTO tells you, that the monthly budget for the data platform is $500, yea we are going to build a cloud native data platform that roughly costs this much

Data Sources – What systems are used to capture the data

With the above information, we then move onto identifying the the systems from which to extract this data from

In Data Engineering terminology, Operations/transaction data is referred to as System of Record (SOR) data.

So, the SOR data sources will be:

• CRM to serve the customer complaints and prospects queries, we note that this runs on a MySQL database
• Accounting System that will serve the CFO & Investment Analyst queries, we also note that it exposes the required source data over a REST API
• Business Support System(BSS) – this is a system (could be built inhouse) custom to our business domain, for a bank, it’s a banking system, for a micro-lending institution, it’s a loan management system (LMS), for a supermarket it’s a Point of Sale (POS). The BSS will serve the Sales, BA queries and runs on a PostgreSQL DB
• If the BSS is behind an ALB(WAF) or API gateway, we will turn on ALB/WAF logging and have them persisted to AWS S3, these logs should address the traffic queries
• The session data will be persisted into a NoSQL storage store

So, we have relational and nonrelational models, our ERP, CRM, and BSS are backed by relational data stores, and these are Postgres DB and MySQL DB

The nonrelational “NoSQL” data stores have 2 categories, Document databases and Graph databases, with document DBs data is represented as self-contained; relationships between one document and another are rare in this case

Graph databases will be used later in this series, but they are used in applications where almost everything is related to anything

Therefore data sources are.