Choosing a news aggregation platform: What to check before you integrate

Every AI provider needs training data. It’s a foundational requirement for AI to learn and provide the most accurate output. In the age of enterprise AI, we are seeing a push to document the provenance of training data. In the past, the most widely used method involved web scraping, a method where bots would crawl a website and extract information which would then be used to train AI models. 

While still an acceptable practice for certain applications, the singular approach of web scraping lacks qualitative value and is now considered to be a legal risk as oversight continues to evolve. For this reason, most AI providers are shifting to using a mix of licensed content and web scraping to ensure brand trust and avoid legal repercussions, while maintaining the speed that is associated with web scraping. 

What web scraping actually gets you and what it doesn’t

While web scraping is continually coming under scrutiny, it remains a largely used practice for collecting large amounts of training data. It is what built the foundation model era, allowing AI to move from small training sets to much larger systems and modern business applications. But as AI continues to gain ground, the open web is quickly becoming exhausted as a source of new, high-quality data. Since most major AI providers already have access to what is openly available, there is no competitive advantage to web scraping in the modern era. 

What web scraping does not get you is access to a steady stream of high-quality data for ongoing purposes. Additionally, it is expected that much of web scraping as a practice may well come under increased fire as tools such as Cloudflare prohibits bots from crawling sites, a practice they enabled on newly created domains after July 2025. Put frankly, scraped content tends to produce weaker AI outputs than licensed content. That’s why data acquisition from licensed sources will become the new norm, especially as what is publicly visible does not mean it can be used for training purposes. 

What licensed data for AI models actually costs

 There is lots of discussion surrounding the affordability of a sustainable, licensed content stream. Most enterprise deals involve confidential terms, making it difficult to get a clear sense of the true market value. In fact, costs vary tremendously when it comes to reliable content. Reports show that Google pays Reddit about $60 million per year, Amazon pays between $20-$25 million a year to The New York Times and OpenAI just closed a deal for content from The News Corp for $50 million per year. 

There is no clear dollar amount that we can yet assign to the cost of content, but what is clear is that the need is only going to grow, making licensed content the only real sustainable path forward. Because it comes with clear provenance and documentation, AI companies will be choosing licensed content to meet regulatory compliance. Although we are still seeing the emerging U.S. fair use guidance take form, the EU AI Act has already made the documentation a legal requirement for AI systems deployed in Europe. 

Why the mix is shifting, not the choice

The truth of the matter is that most AI providers use a blend of web-scraped data along with licensed content and that approach is unlikely to change. What we do see on the horizon is the proportion of this mix leaning more toward the side of licensed content with less reliance on scraped data. Recent copyright litigation is showing more stringent controls will be applied in coming years, making data provenance a key issue.

Anthropic has recently been sued for using copyrighted songs to train Claude models. This comes in the aftermath of a $1.5 billion settlement where the same company paid a group of writers for using copyrighted books train AI. You can see a full list of the active copyright lawsuits on the AI Lawsuit Tracker. The message is clear. There is very real risk in using unlicensed content. While scraping can still be used, high-stakes content is safest originating from licensed content. 

How to build training data provenance into your pipeline

Many organizations are facing the shift to documented data provenance and wondering how to put that into practice. There are some practices that can make this a simpler journey.  

• Match your content source to your goal. If you are performing broad pretraining, you can likely use scraped data to get you started. If you are pushing out customer-facing, journalistic or any other high-stakes content that represents the brand, you need licensed, vetted content. 
• Document the provenance of your content. This may sound overwhelming, but it is quickly becoming standard practice. Document everything including the channel, source, license, terms of use, intended usage, and legal requirement under the EU AI Act. 
• Ensure every data vendor has the appropriate data provenance measures in place. Ask every vendor to confirm how they handle source provenance and tracking, whether they follow robots.txt and platform-level AI signals and how they use licensing for training purposes. These are critical details for any partner.
• Build human review into all sourcing decisions. In a world leaning toward AI, it’s essential that you have a human review process in place for compliance purposes. 

Once you have these items complete, you’ll be able to see how that documentation works in practice.

Why Newtex

In the coming months and years, providers will be evaluating the right recipe of scraped data and licensed sources. While scraped data can be easily sourced, Newstex has licensed, provenance-documented editorial content from a network of publishers, delivered with the documentation this kind of sourcing decision requires to help close the gap.

‍