How to use correctly in coding agents?

[MrPointer]

Summary

Prefix - Discussion Worthy

First of all, I'd like to point out it's not really an issue, probably more of a discussion, but those seems to be turned off for this project, so I have no choice but to create an issue!

Problem

So after the initial, rough start for this project env-wise, I've been giving it a try for a few weeks in both Claude-Code and Opencode, and it seems to me that Claude-Code almost never uses it, at all, let alone correctly or incorrectly. It's evident more by the subagent than the MCP. It seems to always prefer grep/read despite instructing it otherwise - With a mixture of your generated instructions and a bit of my own which try to force it even more.
When questioned about it, it always acts naive, saying that its system prompt tells it to prioritize grep/read over anything else.
So at least in the case of Claude, it looks like semble is meaningless to me, and so my question is - Am I just using it wrong?

I haven't used it as frequently in Opencode, but from the few times I did, it seemed to work a bit better, utilizing the MCP as well (which never happened to me with Claude).

Thanks for the feedback!

[Pringled]

Hey @MrPointer, interesting, this is not something we've encountered ourselves. Couple of questions:

• Which models are you primarily using? We've gotten other feedback that Opus 4.7 has a much stronger preference to grep+read than Sonnet 4.6 for example (which uses Semble without any issues), though we've not had issues ourselves in Claude Code.
• Could you run semble savings and let me know how often it's actually called? It might be that it is being called more than you think. If this is very low there is indeed an issue.
• Do you have other instructions in your AGENTS.md/CLAUDE.md, or other MCPs that could potentially conflict with Semble?

[MrPointer]

Thanks for the reply, @Pringled, let me address each point:

1. I'm mostly using Opus, I admit, whether 4.6 or 4.8 it doesn't matter. With that said, I did configure the semble-search subagent to use Haiku, since it's exploring giant chunks of code anyway. I also tried it without specifying the model, thus inheriting the primary agent's model, but that didn't help either. I'll surely give sonnet a try though!
2. Nice to know that exists! Just ran it now, and it confirms what we feared from - Only 1 call throughout the past week... It had plenty more chances to call it, I can promise you that!
3. Not that I can think of. In fact, I worked hard to tailor them to prefer semble at all costs, and Claude admits it can't get any better than what it already has 😕

[MrPointer]

Update

I did a few more tests, and I can observe the following:

• Sonnet 4.6 seems to respect the instructions to use semble much more than both Opus and Haiku for whatever reason
• semble savings keep showing the same number, even though I've seen it in work during the test... Something doesn't make sense here!

[Pringled]

@MrPointer thanks for sharing! I'll have a look at the semble savings not working properly, that is odd. Are you on Linux, Mac, or Windows? It might be a caching issue. I'll try to debug this a bit more with Opus as well. We will work on proper harness+model evals in the future, but I can't say when we will have that yet. When we have that we will have better ways of quantifying performance per model/harness and hopefully optimize it further.

[MrPointer]

I'm jumping between Mac and Linux currently, it seems both are affected though.

Appreciate the effort to investigate this, I wasn't even sure that semble targets such harnesses as first class consumers, but rather agents themselves, i.e. Agent Developers that create custom agents. Good to know that coding agents are definitely planned and are in fact viewed as a major actor!