Agents and APIs: Sustaining Projects by Rebuilding Them

RRCHNM has been in the business of producing digital history for over thirty years now, and because of this we have developed hundreds of projects on a variety of now-old web software and systems. Five years ago, when I joined RRCHNM, we decided to shift and consolidate our tech stack towards tools and software that would allow us a bit more creativity in the projects we were developing as well as provide easier ways to sustain projects over time. That resulted in a shift in our technical expertise, leading us to prefer the Python-based web framework Django or the static site generator Hugo for projects while reaching for WordPress, Drupal, and Omeka would be used sparingly.

A key task in this shift has been to get WordPress, Drupal, and old versions of Omeka off our servers, which means many of our older projects (and some active ones) had to move to new software. In the past we had simply flattened older sites using wget and either re-hosted them as a flat HTML site or deposited them in George Mason University’s D-space repository. But wget works from a site’s rendered output, which can be a poor picture of what a site actually contains. It sometimes missed content; with Omeka, it grabbed far too much. Older Omeka versions exposed every item not just as a public page but as XML, JSON, and other machine-readable formats, and wget would happily collect all of them, along with every paginated and filtered view it treated as a distinct page. One could tune the crawl with reject patterns, but the underlying problem remained: a project ballooned by thousands of files that played no role in the site’s functioning. Size, of course, isn’t the problem: these extra files were often just plain text and storage is cheap. But it made sites messy and kept files around that we knew users were not using. Plus, we had another problem: if a project intended to keep adding content, we could hardly expect editors to duplicate existing HTML pages and modify them by hand. Static HTML froze a site and made it all but impossible to add new content.

Large language models and agentic programming changed how I could approach this, though not at first in the way I expected. My initial move was to keep the wget method and ask Anthropic’s Claude to run the crawl and then clean up the results, removing the files we knew to be duplicates. This was still intensive work for agents and humans alike. I spent hours trawling through files by hand, or devising regular expressions for find-and-replace that could introduce their own errors—they were never quite precise enough given the loose or absent standards in the HTML of these older projects. It was easy to make mistakes and easy to miss things.

More modern versions of Omeka, WordPress, and Drupal, however, all provide APIs or the ability to expose one. Through their APIs we can access everything that is public in a site: pages, items, author pages, collections, and so forth are all encoded in their APIs for precisely the purpose of machine reading. It dawned on me that I could just give an agent the API and ask it for what I wanted: use the API to find the content, use your bash or browser tooling to get the CSS and HTML layouts, and rebuild this site in Hugo.¹

This process has worked surprisingly well. The first pass of grabbing content via the API is simple: using Python, we crawl the API for pages which are typically categorized by their page type. Paired with something like Python’s BeautifulSoup for processing the HTML, we can very quickly get all of the public content in a site converted into plain text Markdown with appropriate YAML metadata that Hugo expects, plus any customization to that metadata we would want or need (especially if it was already encoded in the API). Then it was a matter of an agent parsing HTML layouts, finding patterns, and grabbing a copy of the CSS. Having all of that on hand allowed the agent to build the template layouts for different page types in Hugo. Finally, capturing all additional content—images, media files, and so forth—was also pulled into the appropriate Hugo directories.

A wget process took a lot of time; paired with an agent and an API, we cut that time down considerably. It i s not unreasonable to have a site completely ready for Hugo within an afternoon and ready for human spot-checking, editing, and cleanup. Agentic-aided sustainability of older sites has addressed many of the problems we were running in to. Rebuilt sites contain only what a user would encounter. And, since content lives in Markdown files rather than frozen HTML, a project that wants to keep growing can simply do so. An agent working against an API cut down the time dramatically: it is not unreasonable to have a site fully converted to Hugo within an afternoon, ready for human spot-checking, editing, and cleanup.