OpenShiftvLLM on OpenShift/ recipes
← All recipes

Meta

Muse Glimmer 30B

Dense 30B multimodal (image-text-to-text) model with a frozen ViT-G/14 perception encoder, served via the RHAIIS muse-glimmer build. Apache 2.0.

30B131,072 ctxmultimodaldense
These manifests assume you have OpenShift with the GPU Operator and RHOAI/KServe installed. Check prerequisites →

Full BF16 across 2× A100-80GB (TP=2). ~60GB weights + vision encoder, leaving headroom for long-context KV cache.

Deploy with oc apply

Save to a file and run oc apply -f deploy.yaml

apiVersion: serving.kserve.io/v1alpha1
kind: ServingRuntime
metadata:
  name: vllm-muse-glimmer-30b
  namespace: llm-serving
  annotations:
    openshift.io/display-name: "Muse Glimmer 30B (bf16)"
spec:
  supportedModelFormats:
    - name: vLLM
      autoSelect: true
  containers:
    - name: kserve-container
      image: quay.io/vllm/rhaiis-early-access:muse-glimmer
      command:
        - python
        - -m
        - vllm.entrypoints.openai.api_server
      args:
        - "--model"
        - "meta-models/Muse-Glimmer-30B"
        - "--tensor-parallel-size"
        - "2"
        - "--tensor-parallel-size=2"
        - "--served-model-name=muse-glimmer"
        - "--enable-auto-tool-choice"
        - "--tool-call-parser=muse_glimmer"
        - "--max-model-len=131072"
      resources:
        requests:
          cpu: "2"
          memory: 8Gi
          nvidia.com/gpu: "2"
        limits:
          cpu: "8"
          memory: 24Gi
          nvidia.com/gpu: "2"
      ports:
        - containerPort: 8000
          protocol: TCP
---
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: muse-glimmer-30b
  namespace: llm-serving
  annotations:
    serving.kserve.io/deploymentMode: RawDeployment
spec:
  predictor:
    model:
      modelFormat:
        name: vLLM
      runtime: vllm-muse-glimmer-30b
      storageUri: hf://meta-models/Muse-Glimmer-30B

Overview

Muse Glimmer 30B is a dense, multimodal (image-text-to-text) causal transformer paired with a frozen ~1.8B ViT-G/14 perception encoder. The language model is 52 layers, hidden size 6656, SwiGLU FFN, GQA (32 query / 2 KV heads), sliding-window 2048, RoPE θ=500,000, vocab 202,048, 131K context. Apache 2.0 licensed.

It is distributed alongside the quay.io/vllm/rhaiis-early-access:muse-glimmer RHAIIS (Red Hat AI Inference Server) build. That image is the standard vLLM OpenAI-compatible server (UBI9 + CUDA 13) with no model baked in — you point it at the meta-models/Muse-Glimmer-30B weights on Hugging Face at runtime.

Prerequisites

  • OpenShift 4.14+ with the NVIDIA GPU Operator installed
  • 2× NVIDIA A100-80GB (or larger) on a single node for tensor parallelism
  • A pull secret with access to quay.io/vllm/rhaiis-early-access (early access)
  • Red Hat OpenShift AI (RHOAI) operator installed, or KServe configured manually

Runtime model download

The image ships with HF_HUB_OFFLINE=1. The repo is public (Apache 2.0), so no HF token is required — just set HF_HUB_OFFLINE=0 in the serving runtime env so vLLM can download the weights on first start. If you mirror the model behind a gated/private repo, also provide HF_TOKEN from a secret.

Quick Start

  1. Save the generated manifests above to deploy.yaml
  2. Apply: oc apply -f deploy.yaml
  3. Wait for the pod to pull the image, download the weights, and start serving
  4. Get the route: oc get inferenceservice

Testing the Endpoint

export URL=$(oc get inferenceservice muse-glimmer -o jsonpath='{.status.url}')
# Text
curl -s $URL/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "muse-glimmer",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'
# Image + text (multimodal)
curl -s $URL/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "muse-glimmer",
    "messages": [{"role": "user", "content": [
      {"type": "text", "text": "Describe this image."},
      {"type": "image_url", "image_url": {"url": "https://example.com/cat.jpg"}}
    ]}]
  }'

Notes

  • Multimodal: accepts text + image input, produces text output.
  • Tool calling: this RHAIIS build ships a dedicated parser, enabled here with --enable-auto-tool-choice --tool-call-parser=muse_glimmer. It is also a reasoning model — add --reasoning-parser=muse_glimmer to separate the thinking trace into reasoning_content.
  • --served-model-name=muse-glimmer exposes a clean model id on the API, independent of the meta-models/Muse-Glimmer-30B repo path.
  • --tensor-parallel-size=2 requires 2 GPUs on the same node plus a large /dev/shm (16Gi here) for NCCL/torch shared memory.
  • BF16 weights are ~60GB; a 4-bit quantized variant is also released upstream if you need to fit on a single GPU (adjust --tensor-parallel-size accordingly).
  • Context length is 131K — reduce --max-model-len if you need to conserve KV cache VRAM.
  • Pin to A100-80GB via the node selector so the pod doesn't land on a smaller or single-GPU node.

References