Quick Overview of Traefik AI Gateway: Kubernetes Deployment & LLM Integration

Traefik's current product offering consists of:

• Traefik proxy (OSS)
• Traefik API Gateway (Enterprise)
• Traefik API Management (Enterprise)

The Traefik proxy can be deployed as an Ingress controller on Kubernetes, free of charge.

API Gateway is a Traefik proxy with neat little authentication extensions, and an AI gateway (see more).

API management is also an enterprise version that enables users to create a centralized API Gateway on Kubernetes. Using declarative definitions (read YAML files), they enable the creation of an API frontend interface. Think of Azure APIM or AWS API Gateway.

API Management includes a simple dashboard to control and overview all API related objects. They also have support for Treblle as a third-party monitoring and tracing dashboard.

To run Traefik API Gateway or API Management, HUB_TOKEN (read license) is needed; otherwise container fails.

Important note

To run the API Gateway or API Management container image to use is given below.

ghcr.io/traefik/traefik-hub

AI Gateway (enterprise only)

Resources for managing AI Gateway using API Gateway:

• AIService
• Middleware
• IngressRoute

Resources for managing AI Gateway using API Management:

• AIService
• Middleware
• IngressRoute
• API
• APIPlan
• APICatalogItem

AI Gateway is just an extension to the TraefikService resource.

Define upstream model (provide key from k8s secret):

kubectl create secret generic antropic-key --from-literal=key=<KEY>

apiVersion: hub.traefik.io/v1alpha1
kind: AIService
metadata:
  name: ai-anthropic
  namespace: traefik
spec:
  anthropic:
    token:
      secretName: "antropic-key"
    model: "claude-3-7-sonnet-latest"

Tie it to the IngressRoute:

apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: gateway
  namespace: traefik
  annotations:
    kubernetes.io/ingress.class: "traefik"
spec:
  entryPoints:
    - websecure
  routes:
    - kind: Rule
      match: Host(`my.gateway.ai`) && PathPrefix(`/`)
      services:
        - kind: TraefikService
          name: traefik-ai-anthropic@ai-gateway-service
          weight: 1
      middlewares:
        - name: content-guard
          namespace: traefik

You can add as many upstream LLM providers, list them under the services, and Traefik will automatically forward the proper request to the selected LLM upstream.

Content Guard? It acts as a Middleware and relies on the external service to implement the guard. Only one is supported, Presidio.

apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: content-guard
  namespace: traefik
spec:
  plugin:
    content-guard:
      engine:
        presidio:
          host: <PRESIDIO_URL>
          language: en
      request:
        rules:
          - jsonQueries:
              - ".messages[].content"
            block: true
            entities:
              - CREDIT_CARD
      response:
        rules:
          - jsonQueries:
              - ".choices[].message.content"
            mask:
              char: "*"
            entities:
              - CREDIT_CARD	

Traefik AI Gateway supports:

• Semantic cache
• Content guard

Interested in other AI Gateways? Read more about the Kong AI gateway, what it offers, and how to deploy it.

Quick overview of the Kong AI gateway

Kong offers multiple editions of its software: * OSS * Enterprise * Konnect (SaaS dashboard for both OSS and Enterprise versions) Konnect is a service for the management and analytics of the Kong Gateways. Konnect offers a control plane and two versions of the Kong deployment: * Managed (Both control-plane and Kong Gateway) * Self-hosted

qdnqnAdnan Selimovic