Smarter Knowledge Bases for Smarter AI Agents

Smarter Knowledge Bases for Smarter AI Agents Web crawling for knowledge bases RAG Improvements Bonus: new Anthropic model available—Claude 3.7 Unlocking more possibilities for AI developers About the author Try DigitalOcean for free Related Articles Announcing OpenAI gpt-oss Models on the DigitalOcean Gradient™ AI Platform Build smarter AI agents: new tools now available for the DigitalOcean Gradient™ AI Platform Introducing GPU Droplets accelerated by NVIDIA HGX H200 By Grace Morgan Published: April 16, 2025 3 min read We’re rolling out new features to the GenAI Platform that make it easier to build, manage, and improve the knowledge bases behind your AI agents. With web crawling, custom crawling rules, and one-click reindexing, you can keep your agents up to date with relevant, real-world information, without manual data collection or external storage. Combined with recent enhancements to our Retrieval-Augmented Generation (RAG) system, these updates can help your AI agents deliver faster, more accurate, and more context-aware responses from richer, better-organized data sources. The quality of your AI agent output depends on the data it can access. With the new web crawling feature, you can crawl publicly available websites and index content into your knowledge base , reducing the need for manual data collection. This is especially valuable for AI agents that rely on public web content to drive insights and actions. Custom crawling rules let you target specific pages or entire domains, ensuring your agent pulls data from the sites you tell it to. One-click reindexing keeps your knowledge base fresh and up to date, making it ideal for use cases like financial analysis and competitive research. No need for external storage , since crawled content is directly integrated into your knowledge base, simplifying setup and reducing overhead. With web crawling, your AI agent can stay informed by pulling in real-time information. To help your agents make the most of that knowledge, we’ve significantly upgraded our Retrieval-Augmented Generation (RAG) system. Accuracy on text-based questions has nearly doubled, reaching up to 95 percent accuracy*.