Artificial Intelligence (AI) has been the buzzword of the last few years. Generative AI (Gen AI) solutions and AI chips brought huge hype and a sense of wonder about what comes next. Companies rushed to put the “AI stamp” on their products or procedures to avoid being considered uninspired or falling behind. However, as a senior analyst at Forrester, Dario Maisto, noted, “There is still an issue of translating this technology into real, tangible economic benefit.”
Does all this mean that the time for creating ChatGPT analogs or copilots for specific industries has passed? Not necessarily. If you can specify the problem and envision how current AI capabilities can help solve it, there is plenty of reason to go after it. To see how this might work, we took a look at the case of web scraping and the industry’s latest AI-based innovation, OxyCopilot — an AI assistant developed by Oxylabs for building scraping and parsing pipelines with minimal coding required.
The trouble with the buzz
The idea that AI has the potential to revolutionize business has been around for far longer than ChatGPT. Nevertheless, the rollout of gen AI tools certainly took it to the next level. AI-related keywords started popping up all over domains that previously saw no need to refer to such tools. Tech startups felt compelled to project themselves as, at least in part, AI startups. While this buzz is understandable, it has at least two major problems.
Firstly, adding AI to your products or business operations does not necessarily give you a new and sustainable competitive advantage. Extracting such an advantage from AI applications might be very difficult unless you already have other advantages to build on, such as hard-to-attain proprietary technology. Meanwhile, training machine learning (ML) algorithms, developing AI solutions, and integrating them into internal software systems are often costly.
This leads to the second issue for businesses and investors – AI-washing. Instead of investing in AI-related research and development, many businesses simply say or imply that AI is at work even when it plays no meaningful role in achieving results. Sometimes, companies will advertise AI capabilities that are seemingly available now. Still, on further investigation, you will learn that it is merely a projection for the future, or, at best, you can contact sales to try an underdeveloped version of a feature that might someday be launched publicly.
Other obstacles to gen AI revolutionizing the markets the way some have hoped include the current technology limitations, such as persistent hallucinations. These noticeable limitations led some to predict that the generative AI hype will slowly decline over the upcoming years. Today, those predictions seem to materialize as investors become disgruntled about the lack of payoff.
Making AI work
At least three things are crucial to making AI work in a way that positively affects ROI and avoids AI-washing. First, one needs to identify the area where AI can have a measurable effect.
In many areas, AI’s potential value lies in automating repetitive tasks. Firstly, this is something AI can generally do quite well. Furthermore, it is a straight path to cutting costs and optimizing resources. Finally, the impact of automation can often be measured and expressed in hard numbers, appeasing shareholders and allowing companies to demonstrate that there is no AI-washing involved.
The second condition is an industry-specific innovation, which already provides advantages to exploit with AI. Building such innovation on already existing proprietary technology allows for amplifying the benefits of that technology. And it becomes a positive differentiator from anyone else who implements AI in the same area. A differentiator that can be pointed out to investors, users, and other stakeholders.
The third requirement for making AI add value is common to many tech industries. Product unification and hygiene enable a reasonable and consistent application of AI. With many scattered tools for particular tasks, billed as separate products, the big-picture that allows seeing the value of adding AI on top of them is hard to grasp.
Case study: AI-powered OxyCopilot
In the case of OxyCopilot, the first-ever AI assistant for web data parsing and request building, all three conditions for making AI add demonstrable value were present. Firstly, parsing was identified as a perfect candidate for a measurable AI boost.
In web scraping, parsing is the structuring of unstructured web data with the help of software tools called data parsers. These tools take time to build and often break down. According to a survey of scraping professionals in the US and the UK, conducted by Oxylabs and Censuswide, 95% of businesses face the negative impact of interrupted parsing within 24 hours.
Developers build data parsers to automatically parse data from specific domains. “The problem is that websites have different HTML structures, meaning separate parsers must be built for each one. This consumes a lot of the developers’ time. Furthermore, websites change over time, causing parsers to stop working, which requires additional resources for maintenance,” said Karolis Kluonaitis, one of the principal developers behind OxyCopilot.
Depending on the complexities at hand, this task can take a developer a few hours, a whole workday, or even more. The Oxylabs and Censuswide study also showed that 86% of scraping professionals find building and maintaining data parsers a time and resource-intensive task, and 49% find it challenging.
“Additionally, 73% of developers have at least tried using AI solutions for web scraping, and of them, two-thirds did it to build parsers. So, even after applying AI to this area, parsing still remains resource-intensive and challenging. Something is lacking,” notes Kluonaitis.
This leads to meeting the second requirement. Oxylabs had previously developed technology for generating templates with instructions for parsing particular domains. This previous technological advantage allowed them to automate large-scale scraping procedures without blowing up the costs of using LLM models when developing AI copilot. Instead of calling LLMs with each request, the copilot can generate parsing templates based on URLs and natural language prompts. Since it can create templates in minutes, the process becomes much faster and scalable to cover all the necessary domains.
Thirdly, Oxylabs unified all its APIs into an all-in-one platform, Web Scraper API, and added an AI-powered copilot as a free feature. According to Kluonaitis, “We made sure that it is as simple to use as possible so that less experienced developers and even non-developers, like managers or support agents with a basic understanding of web scraping could learn to use it.” The unification simplified the user’s workflow, allowing one to clearly see the role of an AI assistant in optimizing the scraping process and the benefits it brings.
In summary: from buzz to value
The developments in gen AI and related technologies offer great potential for businesses. However, making it work to your advantage is not as straightforward as simply adding AI plug-ins wherever seems possible. Businesses should focus on the areas where they already have expertise, technical advantage, or a clear understanding of how AI will add value to go beyond the buzz and benefit from AI applications.
Web scraping is a high-tech task whose usage has grown significantly in recent years and has plenty of potential for future growth. Like many other industries, it sought to adopt AI before and after the gen AI boom started. Thus, AI’s main issues and opportunities, such as deriving demonstrable return on investment and avoiding AI-washing, were exemplified here as clearly, if not better, than in other areas.
The recently released AI copilot for web scraping works because it addresses particular industry issues, has a measurable impact, and builds on proprietary technology. Following this strategy when introducing AI, companies can create sustainable advantages for themselves and their customers.
Disclaimer: GeekWire newsroom and editorial staff were not involved in the creation of this content..
