Modern Solutions Need To Communicate With Each Other
You cannot do business with a customer without communicating; your systems are no different. Application integration and scraping data for custom use is a common struggle. While scraping is essentially just another way to integrate applications, it has its challenges and often needs specialists. Think of it like a call center that needs to support customers that speak another language. Business is modernizing everywhere and forcing even the most ardent defenders of paper filing systems and invoicing into the world of technology. That means your systems need to be able to talk with the systems of vendors, partners, and even customers.
Integration Is Often Over-Simplified
At its core, integration and scraping map data from one system to another. That process should be as easy as reading an address on a letter and placing it in the correct mailbox. Unfortunately, there is far more to consider than moving data from point A to point B. Validations, data types, rules, timing, and data cleanliness must be considered. These issues can sometimes be ignored as outlier issues, but that comes with risks. No one wants their processes to fail at peak business time due to a flawed integration. Therefore, it is better to go into such projects with partners who understand the challenges. That allows them to ask to correct questions and perform the required tests and verification. Data is the lifeblood of modern business and something you should not trust with novices. We have done numerous integrations over the years and take these steps to ensure we provide you with the best solution.
- Start with an overview of your business and the problem or problems you aim to solve. We want to ensure there are clearly understood goals and that we are clear on the “why” of the project.
- Examine current flows, processes, and any related requirements. We want to clearly understand what is needed, where it comes from, and where it will go. The outcome of this step is often a diagram or flow chart of some form. That picture makes it easier for everyone to understand the process and the solution architecture.
- Integration methods are reviewed, and a rough design is created. This may include one or more proofs of concept to verify that the required integrations work. This step often turns up additional requirements and potential data cleansing or transformation that will be needed.
- Data mapping requirements are defined and plugged into the process as a form of test run. This step is sometimes done in two phases. First, the process is run on paper, where each step is validated for the given inputs and desired outputs. Then, a POC is created to push a single record or a small amount of data through the system. A successful run validates the design and mappings.
- The entire process is then implemented with regular UAT or similar data flow validation from source to target.
These steps can be straightforward when you have little data and only two systems to integrate via a stable, well-defined interface. However, they can take months of development time and discussions when dealing with large amounts of data, multiple integrations, fragile interfaces (like scraping), and other factors that add multipliers of complexity.
Trust And Verify
The good and bad of a process is that automation is a multiplier of the results. Thus, automating a valuable process can quickly add a significant amount of value. On the other hand, a process that contains errors will multiply those errors. It is rare to see an integration run flawlessly the first time. Whether the process has flaws or the data itself, integration shines a spotlight on those flaws. A successful integration requires the implementors to understand the impact of the data they are working with and test accordingly. This requirement is even more critical with less stable approaches, such as scraping data from a site. Numerous iterations will be required, and research afterward to ensure that the source and resulting data are exactly as expected. Even a slight variance in the results (or the source can render the entire process (and, thus, the solution) ineffective or, worse, cause data corruption.