The network-request-first approach is the right call. DOM parsing is fragile because it's scraping a rendering artifact — any style refactor, framework upgrade, or A/B test can silently break it. Intercepting the actual API calls the browser is already making is closer to testing the contract, not the presentation.
The healthcare context makes this especially sharp. Those portals are notoriously inconsistent and rarely built for automation. Would be curious how you handle cases where the app uses WebSockets or chunked responses instead of clean REST calls.
Right now libretto only captures HTTP requests, which the coding agent can use to determine how to perform the automation.
For more complex cases where libretto can't validate that the network approach would produce the right data (like sites that rely on WebSockets or heavy client-side logic) it falls back to using the DOM with playwright
The markdown approach has a real advantage people underestimate: you can read and edit the memory yourself. With vector DBs and embeddings the memory becomes opaque — you can't inspect or correct what the model "knows". Plain files keep the human in the loop.
The hard part is usually knowing what +not+ to write down. Every system I've seen eventually drowns in low-signal entries.
This assumes that the model's behavior and memories are faithful to their english/human language representation, and don't stray into (even subtle) "neuralese".
Having run a Markdown memory system with Claude for over a year, I don't think I've seen any evidence of neuralese. That's even with Claude being regularly encouraged to write "reflections" on each session, including automated sessions, and weekly summaries of those reflections.
The bigger problem is avoiding what I call the Memento Effect. I won't spoil the movie for anyone, but Memento involves a character who cannot make new memories, so he has to take meticulous notes about everything. But if any of those notes are vague or incorrect, they still get accept as truth when next reviewed. So you really need your Markdown memory to be pristine and mustn't allow it to become polluted.
The editability is surely an underrated advantage, both for the program itself and the memories it generated.
I think in terms of noise, it is less problematic here because not everything is being retrieved. The agent can selectively explore subsets of the tree (plus you can edit the exploration policy by yourself).
Since there is no context bloat, it is quite forgivable to just write things down.
The healthcare context makes this especially sharp. Those portals are notoriously inconsistent and rarely built for automation. Would be curious how you handle cases where the app uses WebSockets or chunked responses instead of clean REST calls.