Rapid Inference Containers

Rapid Inference Containers (RICs) let you pull a ModelKit as a ready-to-run inference container without a build step. Jozu Hub composes the container at pull time by merging your organization's approved base image with the model weights and metadata from the ModelKit. There is no image to pre-build, maintain, or store. This makes RICs simpler and faster to use and maintain, while being just as secure as pre-built images.

RICs are available in both SaaS and self-hosted deployments. On SaaS, Hub provides four managed container types: vLLM, Llama.cpp, an init container, and a blank container. On-Prem Jozu Hub deployments can define any number of container types, select their own base images, and control which runtimes are exposed to users.

How RICs work

When a container runtime pulls from Hub's RIC endpoint, Hub combines a registered approved base image from your container registry with the model weights from the specified ModelKit in real time. Because both are OCI-compatible, the final image preserves immutability, reproducibility, and layer reuse — just like traditional containers. However, there is no stored artifact that needs tracking or maintenance - the merge exists only in the manifest served to the pulling client.

ModelKits are content-addressable so each has a unique digest that guarantees integrity. When you build or pull a RIC:

• The model and data inside are byte-for-byte identical to the ModelKit
• You can verify that the deployed container uses the exact ModelKit you expect

This ensures safe, auditable, and tamper-proof deployment.

RICs work with any model type: language models, predictive ML, computer vision, NLP, and recommendation engines. They are not limited to specific hardware or GPU vendors.

Pulling an inference container

The base image type is encoded as a path segment in the pull URL. The pull URL follows this pattern:

<registry>/<username>/<repository>/<container-type>:<tag>

For example, to pull the llama-cpp container for a ModelKit tagged 0.5b-instruct:

jozu.ml/jozu/qwen2-0.5b/llama-cpp:0.5b-instruct

The container type segment maps to a registered base image in your configuration. Available types are listed on the Deploy tab for each ModelKit.

The same ModelKit can be served to different runtimes by changing that segment. For example, pulling jozu.ml/jozu/qwen2-0.5b/llama-cpp:0.5b-instruct and jozu.ml/jozu/qwen2-0.5b/vllm:0.5b-instruct from the same ModelKit produces two different containers using two different base images, with the same model weights in both.

Docker

You can get the pull command from the Hub UI or construct it manually.

From the UI: Open the Deploy tab for your ModelKit, select the RIC type and you will see the URL at the bottom of the tab.

From the CLI: Use the modified URL format directly:

docker run -it --rm \
  --publish 8000:8000 \
  jozu.ml/jozu/qwen2-0.5b/llama-cpp:0.5b-instruct-q4_0

The CLI checks for a locally cached image first, then downloads from Hub if needed. Once running, the llama.cpp UI is available at http://localhost:8000/.

Kubernetes

Before you begin, confirm that kubectl is installed and configured to reach your cluster.

Create a namespace:

kubectl create namespace jozu-ric

Create a deployment file (jozu-deploy.yaml):

apiVersion: v1
kind: Pod
metadata:
  name: qwen2-0.5b-llama-cpp
  labels:
    app: qwen2-0.5b-llama-cpp
spec:
  containers:
    - name: llama-cpp-serve
      image: jozu.ml/jozu/qwen2-0.5b/llama-cpp:latest
      ports:
        - containerPort: 8000
---
apiVersion: v1
kind: Service
metadata:
  name: qwen2-0.5b-llama-cpp-svc
spec:
  selector:
    app: qwen2-0.5b-llama-cpp
  ports:
    - protocol: TCP
      port: 8000
      targetPort: 8000

Apply and watch startup:

kubectl apply -f jozu-deploy.yaml -n jozu-ric
kubectl get pods -n jozu-ric --watch

Forward the port to access the model:

kubectl port-forward svc/qwen2-0.5b-llama-cpp-svc 8000:8000 -n jozu-ric

The model is now accessible at http://localhost:8000/. Port forwarding does not expose the service externally. For external access, configure a Service with NodePort or LoadBalancer, use Ingress, or another proxy. You can also use a Deployment resource instead of a Pod to enable horizontal scaling.

Adding a new runtime (self-hosted)

Adding a new base image type as a supported runtime requires one YAML entry in your Helm values. The entry maps a URL path segment to a base image reference in your registry.

Air-gapped deployments (self-hosted)

In air-gapped environments, the RIC server runs locally inside your network and can be configured to point to local registry mirrors for both the base image and ModelKit sources. No connection to external registries is required at pull time.

ArtifactPolicy and digest verification

ModelKits are content-addressable: each has a unique digest that guarantees the model weights and data inside are byte-for-byte identical to the original. When Jozu Agent Guard pulls an inference container produced by the RIC server, it verifies the digest of the ModelKit used to compose it against the attestations stored in Hub. This means the model weights inside a RIC-produced container are subject to the same ArtifactPolicy evaluation as a direct ModelKit pull: a model that fails scan thresholds or lacks the required attestations will not load into the Agent Guard runtime, regardless of how it was packaged.

Limitations

RICs use ModelKit layers as-is, which introduces the following constraints:

• ModelKit contents are mounted into the container root directory
• Files are owned by root with original permissions preserved
• No file transformations or relocations are performed

These constraints can often be addressed by designing containers to work with ModelKits directly, but compatibility issues on certain platforms are possible. Contact [email protected] if you encounter issues.