The 7 concepts of open source models

Tap any concept to jump to its 5-card cluster.

• 1Why Open Source MattersCost, privacy, portability
• 2The Provider LandscapeOllama, LM Studio, Groq, Together
• 3OpenAI-Compatible APIThe universal adapter
• 4Mistral-7B with OllamaDocker for AI models
• 5Provider Swap: Claude → MistralOnly three things change
• 6What Doesn't Port OverClaude-only features
• 7When to Use Open SourceTask-by-task decision

"Right for the cert" is not "right for everything"

This course uses Claude throughout, and for good reason: the cert exam and Claude-only features (extended thinking, computer use, hooks) demand it. But three situations pull developers toward open source anyway.

Cost at scale — 10,000 simple classifications a day on a $40/month VPS beats per-token pricing by an order of magnitude. Data privacy — hospitals, banks, and defense contractors legally cannot send data to a third-party API; local inference is the only path. Portability knowledge — learning the swap tells you which parts of your agent are model-specific and which are yours forever.

Owning the delivery van vs hiring couriers

BEFORE: When you ship a few packages a week, you hire a premium courier. Fast, reliable, no overhead — you pay per delivery and never think about vehicles. That's Claude's per-token API.

PAIN: Now you ship 10,000 identical parcels a day, and some contain records that legally can't leave your building. Paying premium-courier rates per parcel becomes absurd, and handing sealed medical files to an outside driver breaks the law outright. The convenient default is now the expensive, non-compliant choice.

MAPPING: At that volume you buy your own van and driver — a local open source model on your own hardware. Higher upfront effort, but per-parcel cost collapses and nothing ever leaves your loading dock. You pick couriers or the van per route, not once forever.

The three forces, made concrete

1. Cost bends at volumeLow volume: per-token pricing is invisible. High volume of simple tasks: a fixed-cost local model wins big, because the quality gap on easy work is tiny.
2. Privacy is binarySome data can never touch an external API. Local inference keeps it inside your network — a compliance requirement, not a preference.
3. Patterns are yoursRAG, ReAct loops, guardrails, and tracing work identically on any provider. The architecture belongs to you.
4. The positionClaude for reasoning-heavy, agentic, cert work; open source for batch, private, and high-volume-simple work.

Claude, or open source?

# Which model should this workload use?

IF data cannot leave your network:
  USE local open source   # only legal option

ELIF task is simple AND volume is huge:
  USE open source          # cost wins, quality gap tiny

ELIF task is agentic / multi-step
      OR needs cert-only features:
  USE Claude                # reasoning + hooks + caching

ELSE:
  PROTOTYPE both, measure quality vs cost

The choice is per-workload, not a one-time religion. Most real systems use both.

Misconceptions

Four ways to run an open model

Providers differ on where the model runs, who owns the hardware, and what it costs. Four cover almost every case: Ollama (local, free, CLI), LM Studio (local, free, GUI), Groq (cloud, blazing fast, cheap), and Together AI (cloud, hundreds of models, pay-per-token).

The crucial thing they share: all four speak the same OpenAI-compatible dialect, so your code barely changes between them. Ollama is the recommended starting point — no key, no rate limits, nothing leaves your machine.

How you get your electricity

BEFORE: You need power. You could install rooftop solar (own it), buy a plug-in generator (own it, simpler), or just buy from the grid (someone else runs the plant).

PAIN: Pick one blindly and you regret it. Solar is great until a cloudy week; a generator is fine until you need industrial loads; grid power is easy until you need to run off-grid or keep usage private. Each option is right for a different constraint.

MAPPING: Ollama is rooftop solar (self-hosted, private, capped by your hardware). LM Studio is the same, with a friendlier control panel. Groq and Together AI are the grid — someone else's big plant, metered, effectively unlimited capacity. You choose by constraint, not by loyalty.

Who runs what, and when to pick it

1. Ollama — local, freeRuns on your laptop/server. No key, no rate limits, offline. 100+ models. The default for dev and privacy.
2. LM Studio — local, GUIDesktop app for Windows/Mac. Same OpenAI-compatible endpoint once a model is loaded. Good if you prefer clicking to CLI.
3. Groq — cloud, very fastCustom LPU hardware, often faster than Claude, free tier. Use when local hardware isn't enough.
4. Together AI — cloud, many modelsHundreds of models, pay-per-token, plus fine-tuning. Best for experimenting across families.

Which provider for which situation

# Match the situation to the provider

dev on a laptop        -> Ollama    # free, instant, offline
data-privacy required  -> Ollama    # nothing leaves machine
no GPU, need low latency-> Groq      # LPU, ~500 tok/s
exploring many models  -> Together  # 200+ models
Windows, prefer a GUI  -> LM Studio # desktop app
production agentic work -> Claude    # reasoning + caching

RULE: local for privacy/cost,
      cloud when hardware is the limit

Misconceptions

One request shape almost everyone speaks

OpenAI's Chat Completions schema became the de facto standard for talking to LLMs — not by decree, but because so many tools were built around it first. A request is {"model": ..., "messages": [...]}; the reply lives at choices[0].message.content.

Ollama, Groq, Together, and LM Studio all implemented that exact shape. So the same client code hits localhost:11434 (Ollama) or api.groq.com by changing one URL. Anthropic's Messages API is close but different — which is why a raw swap needs a thin adapter.

USB-C ended the cable drawer

BEFORE: Your phone wanted Lightning, your old tablet wanted Micro-USB, your laptop wanted a barrel jack. Every maker chose a plug shape independently, and you carried three cables to be safe.

PAIN: That divergence cost everyone. A charger that fit one device was useless for the next, so you hoarded adapters and still got caught without the right one. Each new gadget with its own port made the mess worse.

MAPPING: USB-C is one plug the whole industry agreed on — charge anything with one cable. The OpenAI schema is USB-C for LLMs: providers adopted it so any OpenAI client "plugs in" by swapping the base URL. The SDK becomes your universal remote.

Three fields that differ

1. System prompt placementAnthropic: a top-level system field. OpenAI: a message with role: "system" at the front of the list.
2. Response pathAnthropic reply is at content[0].text. OpenAI-compatible reply is at choices[0].message.content.
3. max_tokensRequired in Anthropic (no default); optional in the OpenAI schema (falls back to a model default).
4. Everything else matchesRoles, multi-turn history, temperature, and streaming behave the same — so a thin adapter, not a rewrite, bridges the two.

Pseudocode

# ANTHROPIC request
request = {
  model:      "claude-model",
  system:     "You are helpful.",   # top-level
  max_tokens: 256,                # REQUIRED
  messages:   [ user_turn ]
}
reply = response.content[0].text  # Anthropic path

# OPENAI-COMPATIBLE request (Ollama/Groq/...)
request = {
  model:    "mistral",
  messages: [ system_turn, user_turn ]  # system = a msg
  # max_tokens optional
}
reply = response.choices[0].message.content

Misconceptions

Docker for AI models

Ollama is the simplest path to local inference. It handles model download, quantization, hardware detection, and serving behind one CLI. Install once, pull a model like a Docker image, and your laptop becomes an LLM server on localhost:11434.

Mistral-7B at 4-bit quantization needs about 5 GB of RAM and runs on an 8 GB MacBook Air or a 16 GB laptop, using a GPU if one's present and falling back to CPU if not. Tight on RAM? ollama pull phi3:mini is only 2.3 GB.

App store vs building from source

BEFORE: To run a program in the old days you fetched source code, installed the right compilers, resolved dependencies, and prayed it built. Powerful, but a day of yak-shaving before you saw "Hello, world."

PAIN: Running a raw open model is that same nightmare: pick weights, match a runtime, choose a quantization, wire up a server, detect your GPU. Every step is a chance to get stuck, and none of it is the thing you actually wanted to do.

MAPPING: Ollama is the app store — ollama pull mistral and it's installed, quantized, and served. It hides the compiler-and-dependencies pain behind one command, the way an app store hides the build behind "Install."

Install → pull → serve → call

1. Install onceA one-line install script on Mac/Linux, or the Windows installer. Ollama detects your hardware automatically.
2. Pull a modelFetch Mistral-7B (~4 GB, 4-bit) like a container image. It's cached locally for reuse.
3. Serve the APIStart the server; it exposes an OpenAI-compatible endpoint at localhost:11434.
4. Call itHit it with curl, the OpenAI SDK, or LiteLLM — all three behave identically. Connection refused usually just means the server isn't running.

Pseudocode

# one-time setup (shell)
INSTALL ollama
PULL model "mistral"      # ~4 GB, 4-bit
START server              # listens on :11434

# first call — OpenAI-compatible endpoint
POST http://localhost:11434/v1/chat/completions
  body = {
    model:    "mistral",
    messages: [ user("Explain an LLM in 1 sentence.") ]
  }

IF connection refused:
  # server not up -> run `ollama serve`, retry
ELSE:
  reply = response.choices[0].message.content

Misconceptions

Only three things change

Running the same "hello world" against Claude and against Mistral-7B, the concepts are identical — and just three things differ: the client (Anthropic SDK vs OpenAI SDK pointed at Ollama), where the system prompt goes (top-level vs a message), and the response path (content[0].text vs choices[0].message.content).

Two clean strategies: Strategy A uses the openai package with a different base_url. Strategy B uses LiteLLM, where the provider is just a prefix in the model string ("ollama/mistral", "groq/llama3-8b-8192").

Same wall socket, different appliance

BEFORE: Your kitchen wall socket delivers the same power whether you plug in a kettle or a toaster. The wiring in the wall — your app's message loop, history, error handling — never changes.

PAIN: If every appliance needed you to rewire the wall, swapping the kettle for a toaster would be a renovation. That's what it feels like when model-call details leak all through your code — a provider change touches a hundred lines.

MAPPING: Isolate the model call behind one thin function (the socket). Everything upstream plugs into that socket unchanged; swapping Claude for Mistral is swapping the appliance, not rewiring the house. LiteLLM makes the socket truly universal.

Isolate the call, swap behind it

1. Wrap the callPut the model request in one function that takes messages and returns text. Nothing else in your app touches a provider.
2. Branch once insideAnthropic path: top-level system, read content[0].text. OpenAI path: system-as-message, read choices[0].message.content.
3. Point the base URLFor Ollama, an OpenAI client with base_url=localhost:11434/v1 and a dummy API key (Ollama ignores it).
4. Or let LiteLLM routeOne completion() call; the provider is just a prefix on the model name. The loop above stays identical.

Pseudocode

# one function hides the provider
FUNCTION chat(provider, messages, system):
  IF provider == "claude":
    r = anthropic.create(
          system=system,        # top-level
          max_tokens=1024, messages=messages)
    RETURN r.content[0].text
  ELSE:                        # ollama / groq / ...
    msgs = [ system_msg(system) ] + messages
    r = openai_compat.create(model=..., messages=msgs)
    RETURN r.choices[0].message.content

# the loop above NEVER changes when you swap

Misconceptions

The swap covers a lot — but not everything

Basic completions, multi-turn chat, system prompts, temperature, and streaming all port cleanly. But several Claude capabilities have no open-source equivalent: prompt caching, extended thinking, computer use, Claude Code / hooks / sessions, a 200K context window, and native citations.

This isn't a knock on open source — it's the current feature landscape. The good news: your architecture (RAG, ReAct, multi-agent, guardrails, tracing) is model-agnostic and transfers directly. Only the model-specific features stay Claude-only.

Moving countries: skills travel, the SIM doesn't

BEFORE: You relocate abroad. Your language, your professional skills, your relationships — all come with you intact. Those are portable by nature.

PAIN: But your local phone plan, your transit card, your loyalty points, your specific bank's app — those were tied to the old place and simply don't work on arrival. Assuming they'd transfer leaves you stranded at the airport.

MAPPING: Your agent's architecture is your skills — it moves to any model. Prompt caching, extended thinking, computer use, and hooks are the local SIM and loyalty card: platform features bolted to Claude that don't come along. Know which is which before you pack.

What stays behind vs what travels

1. Claude-only platformPrompt caching (cost/latency on long prompts), Claude Code, hooks, and session management are Anthropic platform features — no Ollama equivalent.
2. Claude-only capabilityExtended thinking, computer use, native citations, and a 200K context window have no reliable open-source match today.
3. Tool use is unevenOpen models can do function calling, but reliability varies — Mistral-7B is okay, Llama-3-70B far better. Test before you deploy agentic tool use.
4. Architecture travelsRAG, ReAct, multi-agent, guardrails, tracing, and deployment patterns work identically on any provider.

Does this feature port?

# Before swapping, check each feature you rely on

IF feature is an agentic PATTERN
   (RAG, ReAct, multi-agent, guardrails,
    tracing, deployment):
  -> PORTS cleanly

ELIF feature is a Claude PLATFORM/capability
   (prompt caching, extended thinking,
    computer use, hooks, 200K ctx, citations):
  -> DOES NOT PORT  # rebuild manually or keep Claude

ELIF feature is tool use:
  -> TEST FIRST      # reliability varies by model

Misconceptions

The honest answer is "it depends on the task"

There's no global winner — the right provider is chosen per workload. Claude for complex multi-step reasoning, agentic tool chaining (3+ steps), and cert-specific work. Open source for simple classification/extraction, batch offline processing, data-privacy requirements, and fine-tuning on proprietary data.

The most powerful pattern is hybrid routing: a cheap model (or even a regex) triages requests, and only the genuinely hard ones go to Claude. Mistral-7B as the traffic cop, Claude as the specialist it routes to.

The hospital triage nurse

BEFORE: Imagine every patient walking into an ER is sent straight to the head surgeon, no matter their problem. A paper cut and a heart attack get the same top specialist.

PAIN: The surgeon's time is scarce and expensive. Spending it on paper cuts means real emergencies wait, and the whole system pays a specialist's rate for trivial work — like paying a consultant to press an elevator button.

MAPPING: A triage nurse (a cheap, fast open source model) handles the routine cases and routes only the serious ones to the surgeon (Claude). Route cheap tasks to cheap models; reserve the expensive reasoning engine for the work that actually needs it.

Route by task, not by loyalty

1. Classify the taskSimple/high-volume or complex/reasoning-heavy? Privacy-constrained or not? This decides the lane.
2. Send simple & private work to open sourceClassification, extraction, batch jobs, and anything that can't leave your network go to a local model.
3. Send hard work to ClaudeMulti-step reasoning and agentic tool chaining stay on the frontier model where reliability matters.
4. Add a routerA cheap model triages intent up front and forwards only the complex cases — the hybrid pattern that optimizes cost and quality together.

Task → provider

# Pick a provider per task type

complex multi-step reasoning  -> Claude
agentic tool chaining (3+)    -> Claude
cert prep (M25-M27)           -> Claude

simple classify / extract     -> open source
batch offline processing      -> open source
data-privacy requirement      -> Ollama  # only option
fine-tune on private data     -> open source

BEST: hybrid — cheap model triages,
      Claude handles the hard cases

Misconceptions

Test the core insight

One question per concept. Tap to reveal the answer.