The context window is the maximum number of tokens a model can handle in one request. Input and output combined.

Why this matters:

• Limits how much conversation history you can include
• Determines whether a document fits in one request or needs chunking
• Affects RAG chunk sizes
• Larger windows let the model reason over more information

Current context windows: | Model | Context Window | |-------|----------------| | Claude 3.5 Sonnet/Opus | 200K tokens | | Llama 3.1/3.2/3.3 | 128K tokens | | Llama 3 (8B, 70B) | 8K tokens | | Amazon Nova Pro | 300K tokens | | Amazon Titan Text | 8K tokens | | Mistral Large | 128K tokens | | Qwen3-Coder | 256K-1M tokens |

Token limits cap both input and output sizes.

What you need to know:

• Input tokens: what you send (prompt, context, history)
• Output tokens: what the model generates
• Max tokens parameter: caps response length
• Combined limit: input + output can't exceed the context window

Common limits:

• Output usually capped at 4K-8K tokens per request
• Input limits vary wildly (8K to 300K)
• Embedding models often limited to 8K input
• Exceed the limit and you get a ValidationException

Cost impact:

• You pay per token, input and output priced separately
• Output tokens typically cost more than input
• Larger context = higher cost per request

Claude (Anthropic):

• Complex reasoning and analysis
• Long-form content
• Code review and explanation
• Nuanced conversation
• Strong safety alignment

Llama (Meta):

• Multilingual (8+ languages)
• Code generation
• General-purpose tasks
• Open weights (you can customize)
• Cheaper inference

Amazon Titan:

• Native AWS integration
• Embeddings for RAG
• Enterprise compliance features
• Image generation with watermarks

Mistral:

• Efficient inference
• Strong multilingual
• Mixture-of-experts architecture
• Good at code and reasoning

Hallucinations = plausible-sounding but factually wrong or fabricated content.

Types:

• Factual errors: wrong facts, stats, dates
• Fabrication: made-up citations, names, events
• Conflation: mixing info from different sources
• Logical inconsistencies: contradicting itself

Why it happens:

• Models predict probable tokens, not verified facts
• Training data contains errors
• No real-time fact-checking
• Models don't know what they don't know

How to reduce hallucinations:

1. RAG (Retrieval-Augmented Generation)

   • Ground responses in retrieved documents
   • Model cites sources instead of making things up
   • Primary strategy for knowledge-intensive apps

2. Bedrock Guardrails

   • Contextual grounding checks
   • Validates responses against provided sources
   • AWS claims 99% accuracy for policy checks

3. Prompt engineering

   • Tell the model to say "I don't know" when uncertain
   • Request citations
   • Use chain-of-thought reasoning

4. Human review

   • Review high-stakes outputs
   • Set confidence thresholds for escalation
   • Build feedback loops

Latency = time from request to response.

Metrics that matter:

• Time to First Token (TTFT): how long until the first word appears
• Time to Last Token (TTLT): total generation time
• Tokens per second: generation speed

What affects latency:

• Model size (bigger = slower)
• Input length (more tokens = more processing)
• Output length (longer responses take longer)
• On-demand vs provisioned throughput
• Region (network distance)

Rough latency by model: | Model | Relative Speed | |-------|---------------| | Claude 3 Haiku | Fast (~500ms TTFT) | | Claude 3.5 Sonnet | Medium (~1s TTFT) | | Claude 3 Opus | Slow (~2s TTFT) | | Llama 3 8B | Fast | | Llama 3 70B | Medium |

Throughput = how many requests you can handle.

Bedrock options:

1. On-Demand

   • Pay per token, shared capacity
   • Can get throttled under load
   • Good for variable workloads

2. Provisioned Throughput

   • Reserved capacity, guaranteed model units
   • Lower latency, consistent performance
   • Required for custom/fine-tuned models
   • Higher fixed cost

Limits to know:

• Requests per minute (RPM)
• Tokens per minute (input/output)
• Varies by model and region
• You can request quota increases

Vision (image input):

• Claude 3 models accept images
• Llama 3.2 Vision processes images
• Amazon Nova handles images and video
• Use cases: document analysis, visual Q&A, diagram understanding

Image generation:

• Amazon Titan Image Generator
• Stable Diffusion XL
• Amazon Nova Canvas
• Outputs include watermarks for provenance

Video:

• Amazon Nova Reel generates video
• Nova Pro understands video input

Multimodal embeddings:

• Amazon Titan Multimodal Embeddings
• Text and images in the same vector space
• Enables searching images with text queries