Controllers

Each CRD has a dedicated controller living under internal/controller/. All controllers share the same skeleton:

1. Fetch the CR. If not found, return (it was deleted).
2. Handle finalizers (add if missing; run cleanup if being deleted).
3. Check prerequisites (via RESTMapper — is the upstream CRD installed?).
4. Reconcile managed resources in dependency order.
5. Aggregate and persist status once at the end.
6. Requeue with a strategy appropriate for the phase.

Overview

Controller	Watches	Creates / Manages
PalenaGateway	PalenaGateway	CNPG Cluster, Redis, LiteLLMInstance, NetworkPolicies
PalenaModel	PalenaModel	LiteLLMModel CRs (one per gatewayRef)
PalenaUI	PalenaUI	LibreChat Deployment, MongoDB, MeiliSearch, librechat.yaml
PalenaObservability	PalenaObservability	LangfuseInstance + LiteLLM callback wiring
PalenaMCPServer	PalenaMCPServer	websearch MCP + SearXNG + Chromium + Presidio + FlashRank

PalenaGatewayController

The primary controller. Steps are sequential — each depends on the previous.

Step detail

1. Finalizer: on delete, tear down in reverse order — LiteLLMInstance first, then Redis, then CNPG Cluster, then remove the finalizer.
2. Prerequisites: RESTMapper lookup for Cluster.postgresql.cnpg.io and LiteLLMInstance.litellm.palena.ai. Missing prerequisites → phase: Error, requeue 30s.
3. Database:
   • managed: create/update CNPG Cluster (unstructured), named <gw>-palena-pg, with bootstrap.initdb.database = litellm.
   • external: validate the referenced Secret exists and has the required keys.
4. Redis:
   • managed: create a Secret with generated password (if absent), StatefulSet, headless Service, ClusterIP Service.
   • external: validate Secret.
5. LiteLLMInstance: build the unstructured spec from gateway config, with database + Redis references wired in. See wiring for the exact shape.
6. NetworkPolicies: default-deny + per-path allow rules — LiteLLM → PostgreSQL, LiteLLM → Redis, ingress → LiteLLM.
7. Status aggregation:
   • All components Ready → phase: Running
   • Some not ready → Provisioning / Degraded
   • Any error → Error

Requeue strategy

Condition	Requeue
Error	10s (exponential backoff via controller-runtime)
Component not ready	15s
Success (healthy)	60s periodic health check
Prerequisite missing	30s

Watches

ctrl.NewControllerManagedBy(mgr).
    For(&palenav1alpha1.PalenaGateway{}).
    Owns(&appsv1.StatefulSet{}).
    Owns(&corev1.Service{}).
    Owns(&corev1.Secret{}).
    Owns(&networkingv1.NetworkPolicy{}).
    Complete(r)

CNPG Cluster and LiteLLMInstance are watched via index fields and enqueued using handler.EnqueueRequestForOwner — Palena does not import the upstream Go types.

PalenaModelController

Creates a LiteLLMModel CR for each gatewayRef.

The name of each generated LiteLLMModel is <modelName>-<gatewayName>. After processing all gatewayRefs, the controller lists every LiteLLMModel it owns and deletes those that no longer match a current gatewayRef — this is how ref removals cascade.

Cross-CR watches

PalenaModel also watches PalenaGateway status changes (gateway readiness triggers model sync):

Watches(&palenav1alpha1.PalenaGateway{}, handler.EnqueueRequestsFromMapFunc(
    func(ctx context.Context, obj client.Object) []reconcile.Request {
        // Find all PalenaModels that reference this gateway
    },
))

PalenaUIController

Deploys LibreChat wired to a PalenaGateway, with optional MCP server registration.

Config hash for rolling restart

configHash := sha256.Sum256([]byte(configMapData))
hashStr := hex.EncodeToString(configHash[:])
deployment.Spec.Template.Annotations["operator.palena.ai/config-hash"] = hashStr

Any change to librechat.yaml changes the annotation, which forces a rolling restart — no manual kubectl rollout needed.

Watches

For(&palenav1alpha1.PalenaUI{}).
Owns(&appsv1.Deployment{}).
Owns(&appsv1.StatefulSet{}).    // MongoDB
Owns(&corev1.Service{}).
Owns(&corev1.ConfigMap{}).
Owns(&networkingv1.Ingress{}).
Watches(&palenav1alpha1.PalenaGateway{}, ...).
Watches(&palenav1alpha1.PalenaMCPServer{}, ...)

PalenaObservabilityController

Creates LangfuseInstance CR and wires the Langfuse callback into LiteLLM.

Database sharing strategy

The PalenaGateway creates one CNPG Cluster. When PalenaObservability is added, Langfuse needs its own database in the same cluster. Palena uses CNPG managed.databases to add a langfuse database to the existing cluster, avoiding a second CNPG Cluster.

Callback wiring

Once the LangfuseInstance is Ready and the seed API keys are available:

# Patched onto the LiteLLMInstance spec:
callbacks:
  langfuse:
    enabled: true
    secretRef:
      name: obs-langfuse-apikeys
      keys:
        publicKey: LANGFUSE_PUBLIC_KEY
        secretKey: LANGFUSE_SECRET_KEY
        host:      LANGFUSE_HOST

Every request through the gateway is now traced automatically. On PalenaObservability deletion, the finalizer removes the callback block from the LiteLLMInstance before letting itself be GC'd.

PalenaMCPServerController

Deploys a websearch MCP server with all sidecars. Fully standalone — no gateway reference.

Each sidecar gets its own Deployment + Service, named <mcp-name>-<sidecar>. The MCP server itself mounts a generated config ConfigMap that points at the resolved sidecar endpoints.

Common patterns

Prerequisite checking

func checkCRDExists(mapper meta.RESTMapper, group, kind string) bool {
    _, err := mapper.RESTMapping(schema.GroupKind{Group: group, Kind: kind})
    return err == nil
}

Unstructured CR creation

func buildUnstructuredCR(gvk schema.GroupVersionKind, name, namespace string, spec map[string]interface{}) *unstructured.Unstructured {
    obj := &unstructured.Unstructured{}
    obj.SetGroupVersionKind(gvk)
    obj.SetName(name)
    obj.SetNamespace(namespace)
    obj.Object["spec"] = spec
    return obj
}

Condition helpers

meta.SetStatusCondition(&cr.Status.Conditions, metav1.Condition{
    Type:               "DatabaseReady",
    Status:             metav1.ConditionTrue,
    Reason:             "ClusterReady",
    Message:            "CloudNativePG cluster is ready",
    LastTransitionTime: metav1.Now(),
})

Standard conditions per CRD

CRD	Conditions
PalenaGateway	PrerequisitesMet, DatabaseReady, RedisReady, GatewayReady, Ready
PalenaModel	Synced, Ready
PalenaUI	GatewayReady, MongoDBReady, MeiliSearchReady, ConfigGenerated, Ready
PalenaObservability	PrerequisitesMet, GatewayReady, LangfuseReady, CallbackConfigured, Ready
PalenaMCPServer	SidecarsReady, ServerReady, Ready