SaaS Systems: A Few Pointers on How to avoid Data Acquisition and Integration Problems - One51 Consulting

Written by Julio Acuna | Dec 6, 2020 1:00:00 AM

In 2017, as I witnessed the appearance of Software as a Service (SaaS) systems in the Australian market; and observing firsthand how Organisations of different sizes were adopting those systems, I was questioning the ownership and the availability of the data captured on those systems. Here are those original ideas and update for 2021.

Gone are the days of implementing a single Enterprise Integrated System (a.k.a. ERP) to support all business processes. Instead, organisations opt to use multiple systems (from various vendors), following the premise of “best-of-breed” for each application area.

Regardless of the selected approach for choosing the various systems in an organisation, all of these systems should be implemented according to two fundamental principles for Data Management:

These systems work with data following the workflow of Capture + Process + Store + Report, where Reporting (or a similar method of accessing data) is a must, always.
In an organisation, these systems are always part of a more extensive ecosystem. Therefore, they do not exist in isolation. At some stage, these systems will need to interact with other systems.

Unfortunately, these principles are not always followed, and the post-launch implementation of fixes and workarounds become a project by itself, requiring the additional investment of time and money.

Read the original post here.

5 Common Issues Post-Implementation of SaaS Systems

The following are some of the common issues found by end-users post project implementation. These problems reflect the lack of one or even both of the principles described previously:

The system has been implemented, but it doesn’t provide all the reports we need to run our business operations.
We have access to the system’s data structures, but we cannot make sense of it. Data entities and fields use names we cannot understand, and they don’t reflect the terms used in our business.
We don’t have direct access to the data at all. The only way to extract data from the system is by running a report and then exporting it into a CSV or Excel file.
We want to use a back-end process for system-to-system integration to avoid duplication of effort for manual entry.
The system has the reports we need, but we would like to use those reports’ datasets using a different display format.

How to Avoid or Solve These Issues?

How easy or difficult it is to resolve post-implementation issues will depend mainly on how big the system is, the underlying technologies used to store the data, and the level of customisation used to implement those systems in your organisation.

Like it or not, there is always some dependency on the system’s vendor expertise when wanting to access the database and giving meaning to the data stored in it.

Here are a few tips on how to address the problem of accessing the data and making it available for other use cases:

Ask the system’s vendor to make the data accessible in the shape of a Reporting Database. This database should be well documented and easy to use when data is needed for reporting or for any other downstream systems.
An extension to asking the vendor for a Reporting Database is to ask them to build the additional reports.
Use third-party tools or middleware to abstract the system’s data structures into a metadata layer that reflects your business processes. Use these tools to extract the data into a reporting database and create the reports yourself.
Ask the system’s vendor for API’s or web services you can use to integrate the system with any other internal or external systems. These web services and the Reporting Database must use names that reflect the terms used in your business.
Involve your internal team during and after the project implementation. Working with the vendor will allow your team to learn how to interact with the system’s back-end and where to find the next piece of information requested by the business.
The best advice is always to take early actions to prevent the problem. To achieve this, it is always a good idea to review the overall Enterprise Architecture and how the new system will interact with other systems in the organisation. Also, the precise definition of requirements for data capture, processing and reporting will ensure no surprises by the end of the project.

What’s the situation in 2021?

The use of multiple SaaS (Software as a Service) systems in organisations is now the standard. Names such as Xero, Salesforce, Google Analytics, and similar others are commonly found in all types of organisations. They are like satellite applications or modules interacting with the more traditional SAP or JDE systems.
Most SaaS systems provide REST APIs as the standard mechanism for data transfers in and out of the platform.
A Data Integration approach is a must for teams managing data warehouses. Given that the same data entities exist in multiple systems, the data integration approach ensures the correctness of consolidated datasets.
Some of the technologies used in solving the data acquisition problem include:
- Python is now one of the most used programming languages, and it’s a developer’s favourite tool of coding due to its capability to produce more with less code. Python is a must in the tool arsenal of the now called Data Engineers (modern ETL developers)
- Tools like Fivetran provide an easy and cost-effective way to directly synchronise your data from SaaS systems into your Data Warehouse. Fivetran is the ideal solution when custom coding and the ongoing support for those apps are not possible in your organisation.
- Modern ETL Tools such as Azure Data Factory, Talend or Matillion, to name a few, provide out of the box features for data ingestion from SaaS systems via native connections or REST APIs.

One51 helps Organisations with their data ingestion and data integration needs by using an approach and the technology that make this type of solutions easy to implement and a more cost-effective alternative to the traditional custom ETL development.

View full post