Tag: solutions

  • Data Scraping and System Integration

    Data Scraping and System Integration

    The propensity of data scraping and system integration projects shows we are all figuring out the value of data in our solutions. However, there seems to be a disconnect between this common pursuit and the paths and challenges it offers. The ideas of data scraping and system integration are often seen as very different, but they are the same. We can scrape data from a site or application to integrate with it. Yes, that is typically a one-way integration, but it can go both ways.

    Methods Of Data Scraping And System Integration

    This topic is best started with a list of how to move data from one application to another. Data scraping is one way and likely the most complicated and fragile approach. Therefore, it is essential to consider other options before tackling that “worst-case” option. Integration can be achieved through the following techniques. It is not a comprehensive list but a good overview. For example, we will skip the manual option, which is always available.

    • Web Scraping
    • Direct Link/SSO
    • Application Programming Interface (API)
    • Data feeds (RSS or other)
    • FTP/SFTP and data import/export

    Each option has pros and cons worth examining before you start your next integration project.

    Web Scraping

    This approach is as close to manual integration as you can get. The way it works is the application has a process that connects to the desired system in a way that is identical to a user. For example, the application will open a browser window, enter a URL, send login information, click on buttons, and then copy data from screens and paste it into its data stores. You can mimic a data scraping process by logging into your chosen site, navigating to a desired page, and then writing down values you see on the page. This is a brute-force approach that requires nothing from the integrated system. However, some sites and applications restrict usage and timing. That can frustrate scraping attempts and sometimes is intentional.

    Direct Link/SSO

    This approach is often preferred over scraping, although not much different. We scrape data to pull it from another application into our own. A direct link or SSO configuration can bring the application into our own. The user often sees this as a window within the primary application or as popping open a new window or tab. It can sometimes be a very fluid user experience that is easy to implement. While that integrated application binds the user to its experience, it also will be familiar to them if they have a history of using it separate from your solution.

    Application Programming Interface

    The API solution often is the smoothest way to integrate data between systems. It can be complex and also comprehensive. There are many ways to achieve this, but they can be simplified down to two applications “talking” to each other. While it requires the interface to be defined and stable, it is one of the best ways to create a solution written once and requires little maintenance.

    Data Feeds

    A data feed is similar to an API in many ways. The main difference is that an API can be interactive and done at a data item level. A data feed is a data stream or similar query that can then be processed and pulled into a system. That can be beneficial when real-time integration is not needed. It is often associated with “nightly processing” and can remove the burden of extra processing during peak times of the day.

    Data Import/Export

    A step down from a data feed is an import/export. This can be done on an ad hoc basis and is the equivalent of taking a snapshot of a data block. Data feeds provide a data stream, while an import and export add an integration (import or export the data) and work with files. There are many similarities, but the import/export process does tend to be more static and limited in its use.

    Next Steps

    Feel free to schedule a time to discuss your next integration project with us. Our experience has taught us a lot about the pitfalls and challenges of these project types. Likewise, we have an e-book that can help you explore all the steps in building software, including a few templates. However, we ask that you share an e-mail address so we can send you a copy. We will add you to our monthly newsletter, but you can unsubscribe anytime. Your data is not shared with anyone else. Learn more about our book here.

  • Solving Business Problems, Not An Experiment With The Latest Tech

    Solving Business Problems, Not An Experiment With The Latest Tech

    The modern landscape makes solving business problems an exercise in finding the right provider most of the time. In my experience, software consultants and solutions providers come in three types. Some are heavily invested in existing technology and want to use that for every customer. Then, some want to jump on every new technology and use clients as guinea pigs. The third type focuses on the client and the best solution. Once that solution is defined, they decide how to implement it. These are the ones you want for your project. The challenge is figuring out how to weed through the options.

    Solving Business Problems Is The Focus

    I would say start with the website or proposal. However, you might come across a provider that does not have a website and has not sent you a meaningful proposal. Those are easy to ignore. It is hard to take a technology business seriously that either does not have a website or that has one that looks like it was built by a child a few decades ago. Here are some things a good provider will have on their website. The more of these, the better, and the more details, the better.

    • They talk about successful projects and solutions rather than technology stacks.
    • There are multiple clients represented, whether across projects or stories.
    • The focus is on problems they can solve as opposed to buzzwords.
    • Your non-technical friends, family, and coworkers can understand what they do without searching for the meaning of several terms.
    • There is a start and finish to the projects they highlight.

    While all of the above items are not required, a complete lack of this sort of content should raise warning flags. The alternatives you might see will list a bunch of technology they use (the stack) or provide links to cutting-edge vendors. They likely have the wrong focus when the site looks like an article from a tech news site or program.

    Finding That Perfect Match

    Once you have whittled down your list of prospective providers to those that actually solve problems, your next step is finding a match. The best providers will talk about solving problems you have faced or need to solve. For example, a vendor that spends a lot of time discussing how to market your business may not be the best if you need to track sales or integrate systems. On the other hand, a vendor that talks about bringing customers to your website may be the perfect fit.

    Be Prepared

    This may make you wonder how to match your problem to their skills. That is a good question and does require you to start by ensuring you have correctly understood and defined it. That is a mistake many companies make. They look for a provider and expect the provider to help define the problem and then provide a solution. While that is an acceptable approach, it will likely be more expensive. You will end up in a situation where you are in that old description of paying a consultant to use your watch to tell you what time it is. They are going to be an essential part of crafting a solution. However, you will want to invest the time in defining your problem to solve and related requirements as much as possible.

    A software project can be an expensive process. Think of it like a house or a mansion. You would not go into one of those projects blind or without firm ideas of what to expect. Do the same with your business. Take the time to examine your processes and the pain points you want to reduce or resolve.

    Next Steps

    We have an e-book that can help you explore all the steps in building software, including a few templates. However, we ask that you share an e-mail address so we can send you a copy. We will add you to our monthly newsletter, but you can unsubscribe anytime. Your data is not shared with anyone else. Learn more about our book here.

  • The Value Of A Technology Agnostic Provider

    The Value Of A Technology Agnostic Provider

    There is a concept among technology professionals known as being technology agnostic. This is a valuable trait of providers but is not commonly discussed. It is often avoided because not many providers have this trait. They do not want to highlight a weakness. That prompts a couple of questions right away.

    • What does technology agnostic mean?
    • Why is it a valuable trait?

    I am glad you asked. Let’s set the table with a definition and then look at its value.

    Technology Agnostic Is Broad Vs. Deep

    One of the significant differences between a provider (company or individual) that is technology-specific or agnostic is that the former focuses on one technology. At the same time, the latter has a broad-based approach. For example, a provider might be a Microsoft .NET shop (focused) or an agnostic solution provider. Before we focus on the broad approach, it is worth mentioning that specialists can also be valuable. These are people or teams that live and breathe a specific technology, platform, or application. Sometimes you know exactly what you want, and you want the best. It is not uncommon for a project to get to that point as it is refined and evolves.

    Why A Broad Approach?

    When you start on a project, there are very few constraints. You want to be able to leverage that and get a good understanding of your options. Think of building your dream house. You want to be the one that limits price, material, or location, not your real estate agent. That is the same when solving a problem. Thus, you want to be able to examine all the ways it can be solved without someone arbitrarily making that decision. It is not often highlighted. However, the various technology languages, platforms, and applications have strengths and weaknesses. Therefore, when your provider or consultant is narrowly focused, your options will also be limited. That may be a desire to avoid losing business, or it is often just ignorance. They don’t know what they don’t know.

    An Open Mind Brings Better Solutions

    We all have heard about thinking outside of the box. The best solutions often come when we get out of a box or set of constraints and look at the bigger picture. Technology can be powerful in providing solutions. On the other hand, it can handcuff us and constrain how we approach the problem. Rather than go deep into the weeds, let’s consider standard tools. I will even name names and refer to Microsoft Word and Excel. These are two world-class applications that are used every day. However, they have very different features, strengths, and weaknesses.

    It is entirely possible to write a research paper in Excel. Yet, that is not the best approach. Likewise, one can build a budget solution in Word, but Excel is better. These are almost extreme examples. However, I see such obvious forcing of a solution into a technology daily. It is valuable and possibly critical to start crafting a solution without first selecting a technology, whether an application or a stack.

    Finding The Right Provider

    You might now be asking how you would know what your provider fits into. Are they technology agnostic or a specialist? Fortunately, this is easily solved. You can ask them about their focus, background, and experience in previous projects. A provider constantly referring to one approach or technology will be a specialist. One that lists a large number of technologies is likely agnostic. You can then select them based on whether you know your solution needs their specific talents or you need someone that can guide you to the best technology fit.