What shipped this week - August 21st

I continued to ship skills and build out the platform for Skills and Agents. Some of the highlights:

• New skills: Variance report, W9 vendor check
• UI/UX fixes across all the sites and platforms

Here is how we did the work that shipped

My Eng-Dev Harness got its own repo

The eng-dev harness is the pipeline I use for product building. I use it to plan the work, do the work, review it, check it, and then merge it. The eng-dev process uses a series of agents and was previously run via my Obsidian vault, which also serves as my second brain, with the path hardcoded. There are twelve slash commands, reviewer guidelines for each agent to read before reviewing a diff, and a set of scripts and templates.

One limitation of using the Obsidian vault was how file paths work across multiple machines, so this week I moved the harness to its own repo while keeping the tasks in Obsidian. This adds versioning, pinning like any other dependency, and replaces all hardcoded paths with two environment variables. An install script connects a project's local commands to a specific harness. I split things up so the process itself is code and in git, while task state continues to live in Obsidian, where I can easily read and edit them with states for inbox, planned, in flight, done, the change log, and the cost log.

The practical result is that I can sit down at a different machine and run the same pipeline against the same task queue.

Improved the evals we use to evaluate the skills and the models running the Skills and Agents platform

An eval is a test for the non-deterministic parts of the system, the ones where a model produces the output. A skill runs, and the eval checks that what came back is right.

A passing eval suite doesn't tell you much unless you know the scenarios can fail. A scenario is sharp if it breaks when a specific behavior breaks. If it passes no matter what, it's decorative. Coverage numbers count the same way, so we don't use coverage as a goal here.

We've found that the way to tell the difference is to change something and see if the suite fails intentionally. For evals, our approach is a replay swap: overwrite one scenario's recorded output with another's, then run the suite. If it still passes, the suite can't distinguish between those two scenarios.

This week, we added a CI gate that runs the swaps in addition to the evals we run manually during product development. It only runs when a scenario or replay file is part of the change, since a full sweep is too slow for every pull request. I also added a watchdog, because a stuck step on a self-hosted runner doesn't always get stopped by the built-in timeout.

Model prices

The app shows each organization what every skill and agent run costs in tokens and dollars. The model prices are backfilled based on what we see on the frontier lab platforms.

This week, we added a daily job that fetches OpenRouter's public model list, compares each model we price to it, and records any differences for review. There are two important rules: it never writes to the price table, only reports differences, so a bad upstream number can't change what customers are billed. Also, it matches our model IDs to OpenRouter's using a hand-maintained list, not fuzzy matching, because an incorrect match could lead to comparing the wrong models' prices.

The RAG layer: not yet

The Company Brain is each organization's record of how it really works, including vendors, systems, rules, and edge cases. When the platform reviews a workflow and decides if a step should be automated, it searches the brain and cites what it finds.

The current search scores candidates based on the words in their file names, returns a few results, and then forgets the search happened. One upgrade would be semantic search, which matches meaning instead of just words. That way, a step about "paper invoice approval" could find an entity for Acme Corp even if neither explicitly refers to the other.

I looked into the upgrade this week but decided to hold off on building it. Instead, we decided to add a system to measure search performance. Every retrieval and citation disappears when the request ends, so a suggestion that cited the wrong thing and one that cited nothing look the same afterward. Now the system records every retrieval attempt, what was searched for, what was offered, and what was cited. I then review which brain entity should have been found but wasn't, so any miss is tracked rather than ignored.

We will continue to measure this and see when embeddings are needed. When an organization's brain has reached a certain number of entities, and the miss log shows cases where both description matching and direct search failed but semantic matching would have worked, then we'll consider embeddings. Both conditions have to be met.