Testing

The operator ships with three test levels. Each level has a distinct job — together they cover pure transformations, reconciliation behavior, and real-cluster wiring.

Test levels

Level	What it verifies	Tool	Coverage
Unit	Wiring functions, resource builders, helpers	go test	internal/wiring/, internal/resources/
Integration	Controller reconciliation against a real API server	envtest	All controllers
E2E	Full operator on a real cluster	kind + scripted flows	Critical paths

Unit tests

Unit tests live next to the code they cover (*_test.go). They run in seconds and never touch the network.

Wiring tests

Build* functions are pure — given a Palena CR they return an *unstructured.Unstructured. Tests use table-driven cases to assert on the produced spec map.

func TestBuildCNPGCluster(t *testing.T) {
    tests := []struct {
        name          string
        gateway       *PalenaGateway
        wantName      string
        wantInstances int64
        wantStorage   string
    }{
        {
            name: "basic managed database",
            gateway: &PalenaGateway{
                ObjectMeta: metav1.ObjectMeta{Name: "prod", Namespace: "palena"},
                Spec: PalenaGatewaySpec{
                    Database: DatabaseSpec{
                        Managed: &ManagedDatabaseSpec{
                            Instances:   3,
                            StorageSize: "50Gi",
                        },
                    },
                },
            },
            wantName:      "prod-palena-pg",
            wantInstances: 3,
            wantStorage:   "50Gi",
        },
        // Additional cases: with StorageClass, with PostgreSQL params, with Backup.
    }
    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            cluster := BuildCNPGCluster(tt.gateway)
            assert.Equal(t, tt.wantName, cluster.GetName())
            spec := cluster.Object["spec"].(map[string]interface{})
            assert.Equal(t, tt.wantInstances, spec["instances"])
        })
    }
}

The same pattern applies to TestBuildLiteLLMInstance, TestBuildLiteLLMModel, and TestBuildLangfuseInstance.

Resource builder tests

Resource builders under internal/resources/ return typed Kubernetes objects (*appsv1.StatefulSet, *corev1.Service, etc.). Tests assert on fields directly — names, labels, env vars, volume mounts.

func TestBuildRedisStatefulSet(t *testing.T)   // Name, labels, image, ports, volume claims
func TestBuildLibreChatDeployment(t *testing.T) // Env vars, volume mounts, config-hash annotation
func TestBuildLibreChatConfig(t *testing.T)    // librechat.yaml content (endpoints, MCP, branding)
func TestBuildWebsearchConfig(t *testing.T)    // SearXNG endpoint, sidecar enable/disable, reranker

Helpers

func TestStandardLabels(t *testing.T)
func TestResourceName(t *testing.T)
func TestGenerateRandomString(t *testing.T)

Integration tests (envtest)

envtest is controller-runtime's integration harness — it spins up a real kube-apiserver + etcd locally without kubelet or external controllers. Our controllers run against it exactly as they would in a real cluster.

Setup

var (
    testEnv   *envtest.Environment
    k8sClient client.Client
    ctx       context.Context
    cancel    context.CancelFunc
)

func TestMain(m *testing.M) {
    testEnv = &envtest.Environment{
        CRDDirectoryPaths: []string{
            filepath.Join("..", "..", "config", "crd", "bases"),
        },
        CRDInstallOptions: envtest.CRDInstallOptions{
            Paths: []string{
                filepath.Join("..", "..", "test", "testdata", "crds"),
            },
        },
    }
    // Start env, create client, run tests, stop env.
}

Mock external CRDs

test/testdata/crds/ contains minimal CRD definitions for Cluster (CNPG), LiteLLMInstance, LiteLLMModel, and LangfuseInstance. They have no controllers — envtest only needs the schema so our controller can create CRs of those kinds. We then manually patch their status to simulate upstream behavior.

PalenaGateway flows

func TestPalenaGatewayReconciler_BasicManaged(t *testing.T) {
    // 1. Create prerequisite Secrets (master key, salt).
    // 2. Create PalenaGateway with managed database + Redis.
    // 3. Reconcile → CNPG Cluster CR exists with expected spec.
    // 4. Simulate: patch CNPG Cluster status to Ready.
    // 5. Reconcile → Redis StatefulSet exists.
    // 6. Simulate StatefulSet Ready.
    // 7. Reconcile → LiteLLMInstance exists with correct database/redis refs.
    // 8. Simulate LiteLLMInstance Ready.
    // 9. Reconcile → PalenaGateway.status.phase == Running, all conditions True.
}

func TestPalenaGatewayReconciler_ExternalDatabase(t *testing.T) {
    // external Secret → no CNPG Cluster created, LiteLLMInstance refs the Secret.
}

func TestPalenaGatewayReconciler_MissingPrerequisites(t *testing.T) {
    // CNPG CRDs absent → phase Error, prerequisites.cnpg == false.
}

func TestPalenaGatewayReconciler_Deletion(t *testing.T) {
    // Delete → finalizer tears down LiteLLMInstance, Redis, CNPG Cluster in order.
}

PalenaModel flows

func TestPalenaModelReconciler_MultiGateway(t *testing.T) {
    // Two gateways Ready → one LiteLLMModel per gateway.
    // Remove a gatewayRef → orphaned LiteLLMModel deleted.
}

PalenaUI flows

func TestPalenaUIReconciler_WithMCPServers(t *testing.T) {
    // Gateway Ready + MCPServer Ready (endpoints in status)
    // → librechat.yaml ConfigMap contains the MCP endpoint
    // → LibreChat Deployment env vars reference the gateway Secret.
}

E2E tests

E2E runs against a kind cluster with CNPG, LiteLLM Operator, Langfuse Operator, and the Palena Operator installed.

Minimal deployment

1. kubectl apply -f config/samples/minimal/
2. Wait for PalenaGateway to become Ready
3. Verify PostgreSQL reachable at cluster DNS
4. Verify Redis reachable
5. Verify LiteLLM responding at /health
6. kubectl apply PalenaModel
7. Verify model available via LiteLLM API
8. kubectl delete -f config/samples/minimal/
9. Verify all resources cleaned up

Full deployment

1. kubectl apply -f config/samples/full/
2. Wait for every Palena CR to become Ready
3. Verify LibreChat UI accessible
4. Verify Langfuse UI accessible
5. Verify MCP server responding at /sse
6. Verify LiteLLMInstance has Langfuse callback configured
7. Verify librechat.yaml has MCP server registered

Test utilities

Simulating external CR status

func simulateCRReady(
    ctx context.Context,
    c client.Client,
    gvk schema.GroupVersionKind,
    name, namespace string,
) error {
    obj := &unstructured.Unstructured{}
    obj.SetGroupVersionKind(gvk)
    if err := c.Get(ctx, client.ObjectKey{Name: name, Namespace: namespace}, obj); err != nil {
        return err
    }
    obj.Object["status"] = map[string]interface{}{
        "conditions": []interface{}{
            map[string]interface{}{
                "type":   "Ready",
                "status": "True",
                "reason": "TestSimulated",
            },
        },
    }
    return c.Status().Update(ctx, obj)
}

Waiting for a condition

func waitForCondition(
    ctx context.Context,
    c client.Client,
    obj client.Object,
    condType string,
    timeout time.Duration,
) error {
    return wait.PollUntilContextTimeout(ctx, 1*time.Second, timeout, true, func(ctx context.Context) (bool, error) {
        if err := c.Get(ctx, client.ObjectKeyFromObject(obj), obj); err != nil {
            return false, err
        }
        return conditionIsTrue(obj, condType), nil
    })
}

Running tests

# Unit tests (fast, no env)
make test

# Unit + integration (envtest)
make test-integration

# E2E against a running kind cluster
make test-e2e

# Lint
make lint