Building an Inspect Evals Web Viewer

For the past few months, I've been replicating Quantifying Frontier LLM Capabilities for Container Sandbox Escape with open-source models. Replicating evaluations is pretty straightforward. You open your IDE, download Inspect, then run your tasks.

While the interface gets the job done, working inside an IDE feels like a cluttered experience. I want to see my logs, understand them, and probably have the visualizations ready all in one place. The sole purpose of the Inspect Evals Web Viewer to me was to simply take it as a space to do more of the analytical work outside of the IDE.

To do the same kind of tasks would prompt me to open the IDE, do uv run inspect view, have it opened in another browser, and if I have new logs, I'd have to clear the local database to see a more updated version, then do my analysis there.

Inspect's viewer is pretty comprehensive already. No doubt there. But I propose a way to somewhat wrangle the evals sharing experience. I'm currently in talks with a collaborator and I wanted to share what I ran so far. Usually, a collaborator would just need the copy of the tasks, and they can run it themselves or maybe just share the logs in a Google Drive (this is my first time collaborating on an evals related project, so I don't really know the norms of sharing this type of information lol).

In the case of the project, I'm using AWS EC2s to setup the sandbox for the container escape benchmark. It can be a hassle at times cause of the credentials and infrastructure I had to get setup so that it could follow the same tasks as in the paper. So, I'd figure to just run it myself since it's already working on my end and then just give the evals to my partner.

I guess the sharing part kinda triggered this side project. I saved the logs in a git repository hosted at Codeberg.org and was thinking to just give that repo and be done with it. But I figured they would have to do the whole ordeal of viewing the runs from the IDE and this would affect the speed of delivering a "quick take" on the results of the runs, thus the evals viewer!

I think it's pretty neat and fits my current workflow. I just need to reorganize my logs though because I've mixed some testing on the side (like experimenting on AI gateways because I'm using LLMAPI, Azure and Cloudflare to get AI inference).

It somewhat baffles me that there's limited cloud ways to run evals. I don't know how other people do it (and I guess I haven't talked to a lot of people yet about it lol) but I can't leave my laptop open waiting for my epochs to finish after a few hours.

In a way this is my proof-of-concept of building mini tools to make doing AI safety research more intuitive. If you like what you see, have suggestions or comments, feel free to check it out at https://evals-viewer.nightlybuilds.dev!