package main import ( "database/sql" "encoding/json" "fmt" "os" "time" ) type crObservation struct { Metadata struct { UID, ResourceVersion string CreationTimestamp time.Time Labels map[string]string Generation int } Status struct { Phase string ReadyReplicas int } } func getCR(id string) (*crObservation, error) { b, e := kube("get", "msvc", "r-"+id, "--ignore-not-found", "-o", "json") if e != nil || len(b) == 0 { return nil, e } var cr crObservation if e = json.Unmarshal(b, &cr); e != nil { return nil, e } if cr.Metadata.Labels["admission-id"] != id { return nil, fmt.Errorf("foreign CR") } return &cr, nil } func objectFile(obj any, run func(string) error) error { b, e := json.Marshal(obj) if e != nil { return e } f, e := os.CreateTemp("", "modeldock-object-*.json") if e != nil { return e } defer os.Remove(f.Name()) if _, e = f.Write(b); e != nil { f.Close() return e } if e = f.Close(); e != nil { return e } return run(f.Name()) } // Authorize the worst-case replica count BEFORE handing replicas to HPA. // This lab charges one simulated slot per CPU runtime replica. func elastic(db *sql.DB, id string) error { var max int var uid string e := db.QueryRow("SELECT r.cost,b.uid FROM reservation r JOIN binding b ON r.id=b.id WHERE r.id=? AND r.state='reserved'", id).Scan(&max, &uid) if e != nil { return e } cr, e := getCR(id) if e != nil { return e } if cr == nil || cr.Metadata.UID != uid { return fmt.Errorf("binding absent or replaced") } patch, _ := json.Marshal([]any{map[string]any{"op": "test", "path": "/metadata/uid", "value": uid}, map[string]any{"op": "add", "path": "/spec/autoscale", "value": true}}) if _, e = kube("patch", "msvc", "r-"+id, "--type=json", "-p", string(patch)); e != nil { return e } name := "r-" + id obj := map[string]any{"apiVersion": "keda.sh/v1alpha1", "kind": "ScaledObject", "metadata": map[string]any{"name": name, "ownerReferences": []any{map[string]any{"apiVersion": "lab.modeldock.dev/v1alpha1", "kind": "ModelService", "name": name, "uid": uid, "controller": true}}}, "spec": map[string]any{"scaleTargetRef": map[string]any{"name": name}, "minReplicaCount": 1, "maxReplicaCount": max, "triggers": []any{map[string]any{"type": "prometheus", "metadata": map[string]string{"serverAddress": "http://prometheus.modeldock-lab.svc:9090", "query": fmt.Sprintf("sum(modeldock_waiting_requests{service=%q})", name), "threshold": "2", "ignoreNullValues": "false"}}}}} return objectFile(obj, func(p string) error { _, e := kube("create", "-f", p) if e == nil { return nil } b, ge := kube("get", "scaledobject", name, "-o", "jsonpath={.metadata.ownerReferences[0].uid}/{.spec.maxReplicaCount}") if ge == nil && string(b) == fmt.Sprintf("%s/%d", uid, max) { return nil } return e }) } // Explicit reconciliation. A never-ready, unmodified CR may be compensated only // after its startup deadline. Capacity is returned on a subsequent confirmed absence. func reconcile(db *sql.DB, id string) error { var uid, state string var seen bool e := db.QueryRow("SELECT b.uid,b.ever_ready,r.state FROM binding b JOIN reservation r ON r.id=b.id WHERE b.id=?", id).Scan(&uid, &seen, &state) if e != nil { return e } if state == "released" { return nil } cr, e := getCR(id) if e != nil { return e } if cr == nil { return release(db, id) } if cr.Metadata.UID != uid { return fmt.Errorf("replaced CR; operator investigation required") } if cr.Status.ReadyReplicas > 0 { _, e = db.Exec("UPDATE binding SET ever_ready=true WHERE id=?", id) return e } if !seen && cr.Metadata.Generation == 1 && time.Since(cr.Metadata.CreationTimestamp) > 10*time.Minute { opts := map[string]any{"apiVersion": "v1", "kind": "DeleteOptions", "propagationPolicy": "Foreground", "preconditions": map[string]string{"uid": uid, "resourceVersion": cr.Metadata.ResourceVersion}} return objectFile(opts, func(p string) error { _, e := kube("delete", "--raw", "/apis/lab.modeldock.dev/v1alpha1/namespaces/modeldock-lab/modelservices/r-"+id, "-f", p) return e }) } return nil }