Same experience. I've spent hours tweaking prompts, harnesses, skills and what not to have a simple (but non trivial) project completely managed by an agent.
The key constraint was keeping the quality bar high.
If I lower the quality bar and _only_ care about a working solution, this works great. And I dare not look at the implementation details if this is the outcome I care about.
But if I want a working solution _and_ have it done the right way, then I need to be in the loop at key points. Newer models, unfortunately, have made this problem worse.
I will inevitably come back to some form of bad decisions, deviations or just slop. Left unattended, it snowballs into a pile of mess.
At this point, I decide if I spend time fixing it or switch my mental model to not care about the quality.
The problem is worse when there's a team involved here. Fragmented tooling , prompts, harness, model causes more divergence. Especially in quality. Senior engineers become bottleneck as the quality enforcers.
I do like the revert rate metric that Posthog tracks. This can help incrementally build the tooling to a point it can be trusted and scales.
Author here. Want to share the results of my experiments with local inference. I failed to get decent output from my previous setup (https://news.ycombinator.com/item?id=48587865). But this stack is performing much better.
The local stack only became reliable after fixing multiple separate failures in the stack, fixing the coding harness, improving system prompt and other tweaks in different components.
Concrete repo evidence in the post is from Clawrium(https://github.com/ric03uec/clawrium) PRs #855, #859, and #861. Will continue to iterate on this till I can offload about 30% of the project work completely to the local models.
Author here. Quick context the post doesn't quite spell out:
The tool_choice="auto" failure on Qwen3-Next isn't a parser issue — the model
reasons inside <think>, decides, and never emits the tool call. No error, just
empty tool_calls. The fix was swapping the backbone from Thinking to Instruct,
not tuning any parser flag.
The "load the bigger model first, size the smaller against actual residency"
playbook generalizes to anything with shared CUDA framework overhead. The ~5 GiB
framework floor shows up even at small gpu_memory_utilization values — plan
against actuals, not targets.
Can you try and tune your Claude or whatever LLM you're using for your text to phrase things in plain English. Way less use of antithesis, at least. You can probably find a skill for it, if not get an LLM to write your own.
Yes, there are lots of obvious LLM tells that don't add value, like "the math has to be empirical, not aspirational", use of colorful technical language like "knobs" and "wiring", etc. It distracts from the content.
(...)
- Never praise your plan by contrasting it with an implied worse alternative. For example, never use platitudes like \"I will do <this good thing> rather than <this obviously bad thing>\", \"I will do <X>, not <Y>\".
- Never talk about goblins, gremlins, raccoons, trolls, ogres, pigeons, or other animals or creatures unless it is absolutely and unambiguously relevant to the user's query.
(...)
```
It seems the OpenAI people added that first bullet to specifically address the tendency the model has, as seen in the parent comment. The goblin stuff coincidentally appears right after in the system prompt, so in included it as a bonus.
FWIW the mere fact that the goblin stuff is necessary and is in the system prompt suggests to me that OpenAI's approach of training ridiculously large models that can do everything for everyone is hopelessly cooked.
Though I concede it is not that much different than straightening the tie of your most valuable employee before you unwisely put them in front of a client and saying "please don't tell them about the regressions they didn't notice and remember, they don't want things explained in allegories drawn from the Silmarillion".
Indeed. Maybe in the future labs will be more distinct in what they care about most (and what their model is best at), rather than trying to max out all benchmarks.
This may happen once we see finetuned GLM/Kimi/DeepSeek companies enter the market. I think it's not happening yet because of the hardware supply chain issues.
Building Clawrium (https://github.com/ric03uec/clawrium), a CLI to orchestrate a fleet of autonomous agents(like openclaw, hermes etc) on multiple physical hosts.
As I started running agents on my home network I needed an easy way to maintain them without sshing every every time. I built clawrium as a way to centralize config management, secrets, templates and llm providers. Think of it like a control plane similar to kubectl.
I have 2 OpenClaws running on an old Ubuntu box, and one on a PI. Every config change meant SSHing into each one, tweaking files, restarting services, checking logs. Its all fun during experimentation but now I'm using these for some real work and don't want to spend time troubleshooting config drifts.
So I've been building Clawrium (clm) - a CLI that treats your machines as a fleet and your agents as the things you deploy onto them. Point it at a host, deploy an agent, and you get one command center instead of three terminal tabs.
What works today:
- `clm` CLI to configure instances, LLM providers, IP addresses etc
- End-to-end install + lifecycle management for OpenClaw (Zeroclaw in progress)
- Basic TUI to list all the running agent instances in one place
- Normalized config format; Clawrium translates to Openclaw native format but I intend to add support for all *Claws
What's coming: ZeroClaw, Hermes and other claw support, token-usage guardrails across the fleet, config import/export, backups and other goodies. I'm the solo dev on this so I'm building the features based on my immediate needs.
It's built on Python + Typer + ansible-runner.
Still early and rough around the edges. Would love feedback from anyone running agents across multiple machines, especially on what the CLI ergonomics should feel like.
I'm running a few instances of OpenClaw(and Zeroclaw). I use it for a few things
1. Experimenting with local models. OpenClaw uses local inference on a DGX Spark machine. Helps me understand how different, newer models work and behave
2. Bot 1 helps me maintain a personal project. It sits in discord channel and I throw ideas at it, ask it to log issues, give me priorities, update docs etc. Basically maintenance work to manage the repo
3. Bot 2 is general purpose assistant to take notes(while driving, on a walk, etc), give me daily AI news summary from trusted sources and my relevance criteria. Give me suggestions on what should I dig deeper
4. Bot 3 is setup for work environment. Sits in slack as my team's assistant. Process meeting notes automaticlaly from google and send us summary for the week/month, important notes from a project, answer questions for rest of the org and more. This is most "real" use case so far for me
<plug>
I build https://github.com/ric03uec/clawrium as a lightweight orchestration system to manage agents on a local network because i was having a hard time keeping tabs on these agents. all the ssh-ing and config file gymnastics was too much to handle.
</plug>
I'm a heavy cc user for writing code, reviewing documentation, brainstorming, updating jira tickets etc etc.
For the past few months, I started experimenting with managing a team using cc. As a team, we got together and decided to experiment with a new way to run the team and now that we're looking at some good results, I wanted to share our learnings here
> The problem with the "everyone" model being pitched here is that it may as well be a synonym for "nobody."
Can't agree with this enough!
Thanks for your inputs. A lot of it resonates with what I've observed which translates to the fact that this is as much a cultural/people problem as much it is a technical problem. If teams took ownership by just building visibility, then it'd be an easier problem to solve.
You bring up a good point of doing canary deployments for solving this problem. I'll check this out.
But its interesting that you say ".. if it is a substantial risk in your domain". Isn't this a problem that most engineering teams are struggling with, especially in last few years? Being part of a few DevOps meetups in my area(Seattle) for a while and having attended a bunch of conferences in last couple of year, I've noticed cost coming up as one of the most recurring discussion topics.
Just curious why cloud costs wont be a risk in any domain.
It is a risk for any company, but the possible harm is variable.
At a prior employer, cloud costs could have doubled or even gone up an order of magnitude and because the margins were so good and the tech costs so low, it wouldn't have mattered and may barely have been noticed. Compute wasn't a substantial business cost in any way, as customers were paying for domain expertise in the product.
At another prior employer, costs scaled with revenue pretty linearly, so while bad, it wouldn't be catastrophic before being noticed as it would also mean increased revenue.
However, for say a company that does video streaming where cloud costs are already enormous, poor cloud usage can cut months off runway. Same with AI, where the money is overwhelmingly being burned on compute.
Cloud waste can happen anywhere, but the harm can range from still a tiny number to destroying the ability to make payroll depending on what you are doing.
I agree that building visibility makes accountability easier. Its relatively trivial to build observability for individual services and we have achieved some version of it.
The problem is when 30 odd microservices (each team owning between 5-10) talking to each other. In pre-production setup, changes in few of these services might not have noticeable impact on cost which will become quite apparent in production. When this happens, we definitely notice an increase the cost and the unit metric. But then we dont know where to start fixing this problem from. Right now, this becomes a war-room situation based on the severity but I dont think this is sustainable.
In comparison, if we take API latency as a metric, the accountability and ownership are clearly defined: If an API slows down, the team that owns it, fixes it. They can work with anyone they need to but its their job to fix it.
Did you face similar concerns/issues? Not sure if this is a problem other engineering teams are struggling with or even considering as a real problem to invest it.
I'm also not sure if theres a "standard" way of doing this which we should be thinking about. So, looking for ideas and thoughts here.
I see what you are describing. The pre-production/staging setup may not bring out cost increases caused by application changes, and by the time it is running in production for a while it has already caused a cost explosion.
We did face similar situations - but we fixed them after the cost went up on prod. I guess this has more to do with how much and how fast an "undetected" cost in pre-prod can explode in production. We used to keep an eye on the prod cost numbers after a deployment, and then tackle each one, because the increase was not that quick.
I'm not sure either about a "standard" way, so I'm just thinking aloud here, and I've not tried this myself:
For application changes, measure the difference in cost in pre-prod, in terms of percentage increase, between the previous deployment and the current one, and use that to estimate the possible prod increase. I suspect this will become messy very fast as the other factors to include would be num requests, CPU/memory usage, and so on.
The key constraint was keeping the quality bar high.
If I lower the quality bar and _only_ care about a working solution, this works great. And I dare not look at the implementation details if this is the outcome I care about.
But if I want a working solution _and_ have it done the right way, then I need to be in the loop at key points. Newer models, unfortunately, have made this problem worse. I will inevitably come back to some form of bad decisions, deviations or just slop. Left unattended, it snowballs into a pile of mess. At this point, I decide if I spend time fixing it or switch my mental model to not care about the quality.
The problem is worse when there's a team involved here. Fragmented tooling , prompts, harness, model causes more divergence. Especially in quality. Senior engineers become bottleneck as the quality enforcers.
I do like the revert rate metric that Posthog tracks. This can help incrementally build the tooling to a point it can be trusted and scales.