Limits and pricing

​

Free access

​

Rate limits

1. Stop sending immediate retries.
2. Respect Retry-After if the response includes it.
3. Otherwise wait using exponential backoff with small random jitter.
4. Give up after a limited number of attempts.
5. Show the user a clear, temporary error.

import random
import time
from openai import RateLimitError


def create_with_retry(client, messages, attempts=3):
    for attempt in range(attempts):
        try:
            return client.chat.completions.create(
                model="auto",
                messages=messages,
                timeout=30,
            )
        except RateLimitError:
            if attempt == attempts - 1:
                raise
            delay = (2 ** attempt) + random.uniform(0, 0.5)
            time.sleep(delay)

​

Tokens and context

{
  "usage": {
    "prompt_tokens": 120,
    "completion_tokens": 45,
    "total_tokens": 165
  }
}

​

Keep requests efficient

• Keep system instructions specific and remove repetition.
• Send only conversation history needed for the next answer.
• Summarize older turns in long conversations.
• Use max_tokens when the endpoint and selected model support it.
• Cache deterministic results that are safe to reuse.
• Avoid sending the same request repeatedly after a permanent error.
• Limit concurrent requests in background jobs.

​

Request too large

• The user input
• Included documents
• Conversation history
• Requested maximum output

​

Design for changing capacity

• Set a connection and response timeout.
• Handle 429, 500, and 503 as temporary failures.
• Queue background work instead of starting unlimited parallel requests.
• Track success rate and latency without logging private content or API keys.
• Provide a fallback message when the service is unavailable.