Kubernetes 运维平台

在集群规模扩大后，运维同学频繁执行：kubectl get、logs、rollout restart、临时扩缩容。本平台把只读查询与受控变更封装为 API + 简单 CLI，统一审计、权限与 KUBECONFIG 管理。

---

架构思路

                    ┌─────────────────────────────────┐
                    │         FastAPI Gateway          │
                    │  Auth │ Audit │ Rate Limit      │
                    └───────────────┬─────────────────┘
                                    │
          ┌─────────────────────────┼─────────────────────────┐
          ▼                         ▼                         ▼
   ┌─────────────┐          ┌─────────────┐          ┌─────────────┐
   │  Read Svc   │          │  Change Svc │          │  Event Svc  │
   │ list/describe│          │ rollout/scale│          │ Webhook 订阅 │
   └──────┬──────┘          └──────┬──────┘          └──────┬──────┘
          │                        │                        │
          └────────────────────────┼────────────────────────┘
                                   ▼
                    ┌──────────────────────────────┐
                    │   K8s Client (官方 Python SDK) │
                    │   或 subprocess kubectl       │
                    └──────────────────────────────┘
                                   │
                    ┌──────────────┴──────────────┐
                    ▼                             ▼
              Cluster A (prod)              Cluster B (staging)

设计原则： 读多写少（写操作需 operator + confirm）；按 cluster_id 缓存 Client；变更全量审计；SDK 与 subprocess 按需混用。

---

模块划分

k8s-ops-platform/
├── config/
│   ├── clusters.yaml        # 集群 kubeconfig 路径 / context
│   └── rbac.yaml            # API Key -> 允许的操作
├── k8s/
│   ├── client_factory.py    # 按 cluster_id 构建客户端
│   ├── readers.py           # Pod/Deployment/Event 查询
│   └── writers.py           # rollout restart, scale（受控）
├── services/
│   ├── inspection.py        # 异常 Pod、Pending PVC 汇总
│   └── audit.py             # 审计日志
├── api/
│   ├── routes_read.py
│   ├── routes_write.py
│   └── main.py
└── cli/
    └── ops-cli.py           # typer/click 封装，方便值班脚本

---

核心代码骨架

clusters.yaml

clusters:
  prod:
    kubeconfig: /etc/ops/kube/prod.yaml
    context: prod-admin
  staging:
    kubeconfig: /etc/ops/kube/staging.yaml
    context: staging-admin

client_factory.py

"""按 cluster_id 加载 kubeconfig，缓存 ApiClient。"""
from functools import lru_cache
from kubernetes import client, config
import yaml

def load_cluster_config(cluster_id: str) -> dict:
    with open("config/clusters.yaml") as f:
        data = yaml.safe_load(f)
    if cluster_id not in data["clusters"]:
        raise KeyError(f"未知集群: {cluster_id}")
    return data["clusters"][cluster_id]

@lru_cache(maxsize=8)
def get_apps_v1(cluster_id: str) -> client.AppsV1Api:
    cfg = load_cluster_config(cluster_id)
    config.load_kube_config(config_file=cfg["kubeconfig"], context=cfg["context"])
    return client.AppsV1Api()

writers.py（受控变更）

def rollout_restart_deployment(cluster_id: str, ns: str, name: str, dry_run: bool = False) -> dict:
    """通过 patch restart annotation 触发滚动重启。"""
    from datetime import datetime, timezone
    from k8s.client_factory import get_apps_v1
    apps = get_apps_v1(cluster_id)
    body = {
        "spec": {
            "template": {
                "metadata": {
                    "annotations": {
                        "ops-platform/restartedAt": datetime.now(timezone.utc).isoformat()
                    }
                }
            }
        }
    }
    kwargs = {"name": name, "namespace": ns, "body": body}
    if dry_run:
        kwargs["dry_run"] = "All"
    apps.patch_namespaced_deployment(**kwargs)
    return {"cluster": cluster_id, "deployment": name, "action": "rollout_restart"}

api/routes_write.py

from fastapi import Depends, HTTPException
from pydantic import BaseModel
from services.audit import write_audit
# from api.auth import require_operator

class RestartRequest(BaseModel):
    cluster_id: str
    namespace: str
    deployment: str
    confirm: bool = False  # 防止误触

@app.post("/api/v1/k8s/deployments/restart")  # dependencies=[Depends(require_operator)]
def restart_deploy(req: RestartRequest):
    if not req.confirm:
        raise HTTPException(400, "请设置 confirm=true 以确认变更")
    result = rollout_restart_deployment(req.cluster_id, req.namespace, req.deployment)
    write_audit(action="restart", payload=req.model_dump(), result=result)
    return result

读接口示例：GET /api/v1/k8s/{cluster}/pods、GET /api/v1/k8s/{cluster}/issues（Pending / CrashLoop 汇总）。

---

扩展点

扩展	做法
多集群联邦	cluster_id 路由 + 统一 RBAC；禁止跨集群裸 kubectl
GitOps 协同	写操作仅允许 staging；prod 变更走 MR + Argo CD
巡检报告	定时 list_unhealthy_pods 写入 DB，对接第八阶段巡检平台
Webhook	接收 Argocd / Flux 事件，记录同步状态
资源配额	scale 前读 ResourceQuota，防止 operator 误扩到极限

注意

生产集群不要把 cluster-admin kubeconfig 挂在公网 API 上。使用 RBAC 受限 ServiceAccount，并通过 NetworkPolicy 限制平台 Pod 出站。