In our world, search needs to return results in milliseconds. When I talk to people in content licensing, they tell me that getting results in under a second is great. That gap tells you everything about where this industry has been and where it can go.
At Newstex, we built a Customer Portal where information platforms and enterprise customers can discover, license, and manage content from thousands of publishers, By using modern architecture that brings consumer-tech speed to B2B workflows,we're bringing technology into markets that aren't used to seeing it applied to their problems.
In our world, search needs to return results in milliseconds. When I talk to people in content licensing, they tell me that getting results in under a second is great. That gap tells you everything about where this industry has been and where it can go."
Understanding the legacy landscape
The challenge with legacy systems is they were optimized to do one thing well. They can either handle massive amounts of data, or deliver new content very quickly. Rarely can they do both.
Think about a typical newswire. It's easy to see the story that came in a second ago. But searching for something from three years ago? That takes time. Many of these systems were built when data was backed up to tapes. After a certain point, older content just isn't easily available anymore.
There are also gaps in visibility. When we ask customers for a file ID of a story they're having issues with, they often can't provide it. Not because they don't want to—they just don't have that level of tracking. We track everything that happens, when it happens, and we have instant access to it.
Small companies can adopt emerging technologies much faster than large organizations can. That's where we have an advantage.
The challenge with legacy systems is they were optimized to do one thing well. They can either handle massive amounts of data, or deliver new content very quickly. Rarely can they do both.
Why we went serverless
Our architecture decision came down to one problem: Unpredictability.
We deal with a lot of press releases and associated commentary. During press release season or earnings season we might get 100 times as many stories as on a normal day.
There's no way to predict what will trigger a massive spike. The recent AWS outage triggered a ton of stories. We couldn't have predicted that would happen.
When I first joined Newstex, our database ran on MySQL. It’s a great database that scales well. But for the speed we needed and the inconsistent write patterns we dealt with, we would need to "overprovision", or scale the server up to handle the most traffic we'd expect to get at once. Then we'd have times where we'd have a massive database server that was completely unused. We needed a different approach.
We switched to fully serverless architecture on AWS. We use DynamoDB, a schemaless database that handles our inconsistent data access patterns well. For improved searchability and discoverability, we added TypeSense on top. This is a search library that makes data accessible incredibly fast, in more complex ways than we originally thought possible. This gave us the best of both worlds, a fast query engine for recent data, and a long-term highly scalable system for long-term storage.
Everything runs on Lambda and is entirely serverless. The system automatically scales down when there's no activity and back up with heavy usage. We haven't had scalability pain points since that switch. Even when there was a massive AWS outage, our system immediately scaled up after services were restored and quickly handled the backlog of content. Our system self-healed with very little manual interaction to recover after AWS declared services restored.
Small companies can adopt emerging technologies much faster than large organizations can. That's where we have an advantage.
What this means for customers
Serverless architecture lets us offer something most legacy systems can't: Iinstant access to everything we track, the moment it happens.
Customers can search across years of content, not just recent items. They can drill into details their own systems don't provide. They get a level of granularity that just isn't available in older architectures.
Instead of needing internal teams to acquire and manage all this content, they can come to us as a one-stop shop. Because this is what we do, we've built specific tools to make it efficient.
The serverless stack in action
The entire portal is built on this serverless foundation. Real-time data flows from our databases directly to customers. The frontend uses React because AI coding tools work well with itand the backend is hand-coded for security.
The result brings consumer-tech expectations such as instant search, dynamic filtering, and real-time updates into B2B content licensing.
When a customer who's been in this business for 10 or 20 years tells us this is the best thing they've seen, that validates the approach. We're not making an incremental improvement. We're showing what's possible when you apply modern architecture to these problems.
Setting a new baseline
When a customer who's been in this business for 10 or 20 years tells us this is the best thing they've seen, that validates the approach. We're not making an incremental improvement. We're showing what's possible when you apply modern architecture to these problems.
Being a small company means we can move quickly. We can go serverless when larger organizations are still thinking about it. We can bring millisecond expectations into a one-second world.
That gap I mentioned at the beginning? It's not just a technical difference. It's an opportunity to fundamentally change what customers expect and then exceed those expectations.
About the author,
Christopher Moyer
