How do you decide between a local model and an API for a specific task?

I would use a routing rule based on four questions: is the data sensitive, how difficult is the task, how much latency is acceptable, and what is the cost of a wrong answer?

My default policy would be:

1. Run local for private text, classification, extraction, simple transformations, and short drafts.
2. Use an API for difficult reasoning, broad research, multimodal work, or tasks where the local model fails a small verifier.
3. Escalate only the hard or uncertain cases instead of sending every request to the strongest model.
4. Keep a task-specific evaluation set, because “more capable” on a general benchmark does not mean “better” for your workflow.

This is not just an intuition. RouteLLM trains routers to choose between stronger and weaker models and reports substantial cost reductions without sacrificing response quality in its evaluations. FrugalGPT uses cascades and quality checks for the same cost–quality trade-off.

For multi-step agents I would route at the workflow level as well as per request: a cheap model may be fine for many individual steps but expensive overall if it retries or creates errors that a stronger model would have avoided. The right router should therefore measure completed-task success, retries, latency, and total tokens—not only the price of one call.