Private LLM EU: self-host or Claude on Bedrock 2026

TL;DR

A private LLM in the EU is a hosting choice, not a brand. Weights sit on your GPUs, in Frankfurt through Bedrock Custom Model Import, or you call Gemma 4 in eu-central-1. Claude stays behind eu.anthropic.claude-sonnet-5-5. The Anthropic API has no eu value. Foundry documents Global Standard and US Data Zone only.

What counts as a private LLM in the EU?

A private LLM in the EU is a model whose inference runs under your control in a member state or in documented EU geography. It is not a chatbot plan. Three layers matter: where the weights live, where the request is computed, and who stores prompts. GDPR does not require an EU location. Residency still removes the transfer question.

This is not legal advice. The provider matrix is in Claude GDPR compliance in Europe. How to pin Claude itself is in Claude data residency in the EU.

Private does not mean open source. Gemma 4 and Qwen3-32B are Apache 2.0. So is Mistral Large 3. Llama 4 Scout uses the Llama 4 Community License with a 700 million monthly active user threshold. DeepSeek-V4.1-Flash is MIT. Claude weights stay closed.

Layer	What you check	Typical trap
Weights	Licence, parameters, context	Llama MAU threshold, 200 GB text import cap
Inference location	Region, geo profile, your GPU node	inference_geo is only global and us
Storage	Logs, ZDR, aws_review	Fable 5 and 5.1, up to 30 days inside AWS

The Bedrock data protection page states that model providers have no access to deployment accounts, logs, prompts or completions. That covers Claude and open-weight models on Bedrock. Self-hosting gives you the same lever without AWS in the middle.

Which three paths exist in 2026?

A private LLM in the EU in 2026 is one of three setups. You self-host weights on hardware you run. You import customised weights into Frankfurt, or you call Gemma 4 and Qwen3 in a single EU Region on Bedrock. Claude remains the quality default behind the EU geo profile. Foundry does not pin Claude to an EU zone.

Path 1: self-host. The Gemma 4 31B-it card lists Apache 2.0, 30.7 billion parameters, 60 layers, 256K context. Qwen3-32B is Apache 2.0, 32.8 billion parameters, 32,768 tokens natively, 131,072 with YaRN. Mistral Large 3 is Apache 2.0, 675B total, 41B active, 256k context, vLLM from 1.12.0 on one 8xH200 node. DeepSeek-V4.1-Flash is MIT, 552B backbone, 8B prefill and 16B decode, 1M context. GPU sizing for Flash is in DeepSeek V4.1 Flash self-hosted.

Path 2: managed open-weight on Bedrock. Gemma 4 31B uses ID google.gemma-4-31b, launch 31 March 2026, EOL no sooner than 31 March 2027, bedrock-mantle only, no geo and no global profile. Frankfurt is the documented EU row. Qwen3 32B uses ID qwen.qwen3-32b-v1:0, launch 29 April 2025, EOL no sooner than 29 April 2026, 32K context, 8K output, in-region including Frankfurt, Stockholm, Milan, Ireland and London. Custom Model Import runs in eu-central-1, us-east-1, us-east-2 and us-west-2. Text weights must stay under 200 GB, multimodal under 100 GB, context under 128K. Supported architectures include Llama 2 through 3.3, Mistral, Mixtral, Qwen2/Qwen2.5/Qwen3, GPTBigCode. GPT-OSS import is N. Virginia only. The catalogue is in What is Amazon Bedrock.

Path 3: Claude with EU pinning. The Sonnet 5.5 card lists launch 28 September 2026, EOL no sooner than 28 September 2027, 1 million tokens of context, 128,000 output, June 2026 cutoff. The EU profile eu.anthropic.claude-sonnet-5-5 routes from Frankfurt to Frankfurt, Stockholm, Milan, Spain, Ireland and Paris. London is an extra source, not a destination from Frankfurt. Google keeps Sonnet 5.5 in Europe multi-region at 1,250 QPM, 12,500,000 uncached input TPM and 1,250,000 output TPM. The EU host is aiplatform.eu.rep.googleapis.com, location eu. The United Kingdom and Switzerland are excluded. First-party inference_geo accepts only global and us. Workspace geo is us only. Foundry lists Global Standard and Data Zone Standard (US). No EU data zone for Claude is documented there.

We recommend path 3 as the default when the job needs agents, computer use, Guardrails or a million tokens of context. Path 2 when the dollar rate wins and in-region Frankfurt is enough. Path 1 when the weights must sit in your data centre.

The model-by-model pages stay separate: Gemma vs Claude, Mistral vs Claude, Llama vs Claude, Qwen vs Claude.

What does a private LLM cost versus Claude?

Token rates sit on the vendor pages. Claude Sonnet 5.5 lists 2 / 10 USD per million tokens, Opus 5.5 lists 4 / 20. Regional endpoints from Sonnet 4.5 add 10 percent, so EU Sonnet is 2.20 / 11. Gemma 4 31B in Frankfurt lists 0.17 / 0.48. Custom Model Import in Frankfurt lists 0.07144 USD per unit per minute.

The Anthropic pricing page lists Sonnet 5.5 cache reads at 0.10 USD. Haiku 5.5 up to 100,000 tokens costs 0.10 / 0.50. Batch halves input and output. A CCU is 0.01 USD, 100 CCU equal 1 USD. US-only inference_geo on the first-party API is 1.1x. Bedrock Priority is 75 percent above Standard. Flex and Batch are 50 percent below.

Path	Input / output per 1M tokens	Residency	Ops
Claude Sonnet 5.5 global	2 / 10 USD	anywhere	no GPU
Claude Sonnet 5.5 EU geo (10 %)	2.20 / 11 USD	EU destinations on the card	no GPU
Gemma 4 31B US	0.14 / 0.40 USD	one US Region	Mantle /openai/v1
Gemma 4 31B Frankfurt	0.17 / 0.48 USD	eu-central-1	Mantle /openai/v1
Custom Model Import Frankfurt	0.07144 USD per unit per minute, storage 1.95 USD per unit per month	eu-central-1	scale to zero after 5 minutes idle
Llama 3.1 8B import (reference)	2 units	Frankfurt or US	as import
Llama 3.1 70B import (reference)	8 units	Frankfurt or US	as import

Two units in Frankfurt cost 0.14288 USD per active minute. Eight units cost 0.57152 USD per active minute. Billing runs in 5-minute windows from the first successful invoke. After five idle minutes Bedrock scales to zero. The default cap is 3 copies per imported model, raisable through Service Quotas. We did not read EU GPU rental rates on an allowed source in this run, so we leave them blank.

The API table across Bedrock, Google and Foundry is in Claude API pricing EU.

Which licences and AI Act roles apply?

Licence and the AI Act are separate tests. Apache 2.0 and MIT allow commercial use with attribution. The Llama 4 Community License wants “Built with Llama” and an extra licence above 700 million MAU. The AI Act follows the role, not the licence. High-risk duties for Annex III systems apply from 2 December 2027.

The EUR-Lex summary lists general application from 2 August 2026, AI literacy and bans from 2 February 2025, GPAI provider duties from 2 August 2025. Omnibus Regulation (EU) 2026/1744 of 8 July 2026 moves Chapter III sections 1 to 3: Annex III from 2 December 2027, Annex I from 2 August 2028.

Calling Gemma or Qwen keeps you a deployer of the system. Fine-tuning, importing or offering a service to others can also make you a provider. Roles are in EU AI Act for LLM deployers. Fable 5 and 5.1 on Bedrock require aws_review: prompts and completions stay up to 30 days inside the AWS boundary, the model provider does not see them. Most other models allow none. If you set data_retention_mode to none, Bedrock blocks models that require retention.

How do you set the EU default?

Pinning is an endpoint, not a policy PDF. Claude only through eu.anthropic.claude-sonnet-5-5 or Google locations/eu. Gemma 4 only through bedrock-mantle.eu-central-1.api.aws/openai/v1. Self-host Qwen3-32B with vLLM. Block global. in IAM. Foundry still has no EU zone for Claude, so it is not a default.

1. Write residency into the brief: one member state, EU geography or global.
2. Pick the path: Claude EU, Bedrock in-region open-weight, import or your own node.
3. Lock the model ID and re-read the card on the day you ship.
4. Set retention to none unless Fable is in scope on purpose.
5. Cap spend on a gateway so apps do not fall through to us-east-1.
6. Record licence, AI Act role and EOL.

import boto3
from openai import OpenAI

# Claude EU geo via Converse
br = boto3.client("bedrock-runtime", region_name="eu-central-1")
claude = br.converse(
    modelId="eu.anthropic.claude-sonnet-5-5",
    messages=[{"role": "user", "content": [{"text": "State the residency limits in two sentences."}]}],
    inferenceConfig={"maxTokens": 256},
)
print(claude["output"]["message"]["content"])

# Gemma 4 31B in Frankfurt, Mantle /openai/v1 only
gemma = OpenAI(
    api_key="BEDROCK_API_KEY",
    base_url="https://bedrock-mantle.eu-central-1.api.aws/openai/v1",
)
print(gemma.chat.completions.create(
    model="google.gemma-4-31b",
    messages=[{"role": "user", "content": "State the residency limits in two sentences."}],
).choices[0].message.content)

# Path 1: Qwen3-32B on your network (vLLM per the model card, vllm>=0.8.5)
vllm serve Qwen/Qwen3-32B --enable-reasoning --reasoning-parser deepseek_r1

Mistral Large 3 on one 8xH200 node:

pip install 'vllm>=1.12.0'
vllm serve mistralai/Mistral-Large-3-675B-Instruct-2512 \
  --max-model-len 262144 --tensor-parallel-size 8 \
  --tokenizer_mode mistral --config_format mistral --load_format mistral \
  --enable-auto-tool-choice --tool-call-parser mistral

We recommend Claude on Bedrock EU as the default in European companies. A private LLM sits beside it, not instead of it, when weights, price or an import workflow win the brief. Run both behind one gateway with separate limits.

FAQ

What does a private LLM in the EU cost versus Claude?

Gemma 4 31B in Frankfurt lists 0.17 / 0.48 USD per million tokens. Claude Sonnet 5.5 lists 2 / 10, with a 10 percent premium on regional endpoints: 2.20 / 11. Custom Model Import in Frankfurt lists 0.07144 USD per unit per minute plus 1.95 USD storage per unit per month. We did not verify GPU rental rates here.

Private LLM vs Claude: which should we take?

Claude Sonnet 5.5 behind eu.anthropic.claude-sonnet-5-5 when personal prompts must stay in member states and you need agents, computer use or 1 million tokens of context. A private LLM when Apache 2.0 weights, the Frankfurt rate or an import into eu-central-1 win the brief. Foundry does not replace either path for EU pinning of Claude.

Does Gemma 4 stay in the EU on Bedrock?

In Frankfurt, yes, as in-region on bedrock-mantle, with no geo profile. The card does not support geo or global. The request body including images and video must stay under 3.5 MB. Reasoning tokens come back on the Responses API, not Chat Completions.

Can a European company run Llama 4 Scout privately?

Yes, under the Llama 4 Community License, as long as the 700 million MAU threshold does not apply and the Acceptable Use Policy holds. Scout has 17B active, 109B total, 10M context. On Bedrock, the Llama 4 cards we read have no EU geo. Import of Llama 3.x into Frankfurt is documented. Llama 4 is not on the import architecture list.

Does self-hosting make us an AI Act provider?

Calling the model keeps you a deployer. Fine-tuning, re-releasing or offering a service to third parties can add provider duties. GPAI model duties apply to the original provider from 2 August 2025. High-risk systems under Annex III from 2 December 2027. Settle the role with your compliance team, not with this page.

Do Anthropic or Google see our prompts on Bedrock?

AWS says no. Model providers have no access to the deployment accounts. AWS may retain data for abuse detection. Fable 5 and 5.1 require up to 30 days of aws_review inside the AWS boundary. Self-host if AWS must not see the prompts either.

Sources

1. Anthropic: Pricing (19 October 2026)
2. Anthropic: Data residency (19 October 2026)
3. AWS: Claude Sonnet 5.5 Model Card (19 October 2026)
4. AWS: Gemma 4 31B Model Card (19 October 2026)
5. AWS: Qwen3 32B Model Card (19 October 2026)
6. AWS: Custom Model Import (19 October 2026)
7. AWS: Data protection (19 October 2026)
8. AWS: Data retention (19 October 2026)
9. Amazon Bedrock Pricing (19 October 2026)
10. Hugging Face: gemma-4-31B-it (19 October 2026)
11. Hugging Face: Mistral-Large-3-675B-Instruct-2512 (19 October 2026)
12. Hugging Face: Qwen3-32B (19 October 2026)
13. Hugging Face: Llama-4-Scout-17B-16E-Instruct (19 October 2026)
14. Hugging Face: DeepSeek-V4.1-Flash (19 October 2026)
15. Google Cloud: Data residency (19 October 2026)
16. Google Cloud: Claude Sonnet 5.5 (19 October 2026)
17. Google Cloud: Use partner models (19 October 2026)
18. Microsoft Learn: Claude models in Foundry (19 October 2026)
19. EUR-Lex: Regulation (EU) 2026/1744 (19 October 2026)
20. EUR-Lex: AI Act summary (19 October 2026)