Topics / AI engineering & agents
AI engineering: agents that ship work, not demos
Operators in the club run coding agents and LLM workflows every day. The conversation has moved from which model is smartest to how to wire agents into repositories, route requests across models and keep token spend and context under control.
In the club
Counted across 255,431 messages from 1,844 members. Nobody is named or quoted.
Tools discussed
- ChatGPT / OpenAI2,345
- Claude / Anthropic989 ↑
- Cursor297
Rising threads of conversation
Model routing to manage cost
35.9/1k now vs 20.5/1k before
Context and memory for agents
20.4/1k now vs 12.8/1k before
Browser agents for GTM work
27.8/1k now vs 18.5/1k before
What people ask for help with
- AI & agent reliability / setup700%
Share of 497 sampled questions
Why it matters
Agent reliability and setup is one of the problems members raise most. The teams getting value treat agents like any other production system: scoped tools, cheap models for easy steps, evals, and humans on the risky decisions.
What practitioners should know
- 01
Harnesses matter as much as models
The same model behaves very differently depending on the harness: its tools, instructions, memory and how it recovers from errors.
- 02
Route by task
Cheap fast models handle classification and extraction; frontier models take planning and hard reasoning. Routing keeps cost predictable.
- 03
Context is a budget
Long contexts are expensive and dilute attention. Members trim what goes in, cache what repeats and keep notes outside the window.
- 04
Evaluate before trusting
Small task-specific evals catch regressions when models, prompts or tools change underneath a workflow.
Worth reading
- Harness Engineering: Anatomy, Architecture, and Evolution of Coding Agents - A Source-Code Study of Eleven Systems
A source-code study of eleven coding-agent harnesses that catalogs recurring design patterns and argues the coding harness shifted from tool to platform in the first half of 2026.
arxiv.org
- Build an Agent Improvement Loop with Traces, Evals, and Codex
OpenAI cookbook walkthrough for improving an Agents SDK agent by collecting traces, running evals and using Codex to propose fixes in a repeatable loop.
developers.openai.com
- State of Agent Engineering
LangChain survey of practitioners on how agents are deployed in production, the main quality barriers, and adoption of observability versus evals.
langchain.com
- Building AI Coding Agents for the Terminal
Paper describing a model-agnostic terminal coding agent built as a compound AI system, with per-role model binding, context compaction and separate plan and execute modes.
arxiv.org
- Free the models: Harness design at the frontier | Replit
Replit reports that giving frontier models freedom to delegate to cheaper subagents outperformed more constrained harness designs in its experiments.
replit.com
Research gathered Sep 29.
Questions to be able to answer
- Which steps in your agent loop could run on a cheaper model?
- How do you know when a prompt or model change broke a workflow?
Public threads on this
- I built an open leaderboard for non-neural AI looking for systems and benchmark feedbackr/LLMDevs
- Is symbolic ai dead? [D]r/MachineLearning
- Excited about AI agents, but also slightly terrified of letting them loose in my business!r/AI_Agents
- The same AI model gets added, repriced and retired on different dates on Bedrock, Azure and Vertex, so I built change alerts for all threer/LLMDevs
Club numbers as of Oct 24. Words are reviewed before they go live. Links go to their publishers.