Llmisvc Model Provider Resolver

Rewrites publisher-ID body model values to the short model name for LLMISvc / KServe routing; the routing header (default X-Model) is left unchanged so routing can still use the publisher ID

Category: Setup-dependent integration
Task: Rewrites publisher-ID body model values to the short model name for LLMISvc / KServe routing; the routing header (default X-Model) is left unchanged so routing can still use the publisher ID

Prerequisites: The external service, credentials, or certificates referenced by this configuration.

Run it: Use ghcr.io/praxis-proxy/ai:0.5.0 and follow the container quickstart to mount and start the configuration.

This configuration comes from the selected release. The example has not been run here; external services are not bundled.

Download the source file.

# LLMISvc Model Provider Resolver
#
# Build:
#
# Rewrites publisher-ID body `model` values to the short model name for
# LLMISvc / KServe routing; the routing header (default `X-Model`) is
# left unchanged so routing can still use the publisher ID.
#
# Typical chain: `model_to_header` promotes the body model to
# `X-Model`, then `llmisvc_model_provider_resolver` strips the
# publisher prefix from the body only.
#
# Example request:
#
#   curl -X POST http://localhost:8080/v1/chat/completions \
#     -H "Content-Type: application/json" \
#     -d '{"model":"publishers/rhoai/models/granite-3.1-8b","messages":[{"role":"user","content":"hi"}]}'
#
# Upstream body model becomes `granite-3.1-8b` while `X-Model`
# remains `publishers/rhoai/models/granite-3.1-8b`.

listeners:
  - name: llmisvc-gateway
    address: "0.0.0.0:8080" # dev value; binds all interfaces
    filter_chains:
      - rewrite-and-route

filter_chains:
  - name: rewrite-and-route
    filters:
      - filter: model_to_header
        header: X-Model
      - filter: llmisvc_model_provider_resolver
        header: X-Model
      - filter: router
        routes:
          - path_prefix: "/"
            cluster: provider
      - filter: load_balancer
        clusters:
          - name: provider
            endpoints:
              - "127.0.0.1:3000"

insecure_options:
  allow_private_endpoints: true # example proxies to local backends