add brain

This commit is contained in:
2026-03-12 15:17:52 +07:00
parent fd9f558fa1
commit e7821a7a9d
355 changed files with 93784 additions and 24 deletions

View File

@@ -0,0 +1,48 @@
# CI/CD Pipeline Builder
Detects your repository stack and generates practical CI pipeline templates for GitHub Actions and GitLab CI. Designed as a fast baseline you can extend with deployment controls.
## Quick Start
```bash
# Detect stack
python3 scripts/stack_detector.py --repo . --format json > stack.json
# Generate GitHub Actions workflow
python3 scripts/pipeline_generator.py \
--input stack.json \
--platform github \
--output .github/workflows/ci.yml \
--format text
```
## Included Tools
- `scripts/stack_detector.py`: repository signal detection with JSON/text output
- `scripts/pipeline_generator.py`: generate GitHub/GitLab CI YAML from detection payload
## References
- `references/github-actions-templates.md`
- `references/gitlab-ci-templates.md`
- `references/deployment-gates.md`
## Installation
### Claude Code
```bash
cp -R engineering/ci-cd-pipeline-builder ~/.claude/skills/ci-cd-pipeline-builder
```
### OpenAI Codex
```bash
cp -R engineering/ci-cd-pipeline-builder ~/.codex/skills/ci-cd-pipeline-builder
```
### OpenClaw
```bash
cp -R engineering/ci-cd-pipeline-builder ~/.openclaw/skills/ci-cd-pipeline-builder
```

View File

@@ -0,0 +1,147 @@
---
name: "ci-cd-pipeline-builder"
description: "CI/CD Pipeline Builder"
---
# CI/CD Pipeline Builder
**Tier:** POWERFUL
**Category:** Engineering
**Domain:** DevOps / Automation
## Overview
Use this skill to generate pragmatic CI/CD pipelines from detected project stack signals, not guesswork. It focuses on fast baseline generation, repeatable checks, and environment-aware deployment stages.
## Core Capabilities
- Detect language/runtime/tooling from repository files
- Recommend CI stages (`lint`, `test`, `build`, `deploy`)
- Generate GitHub Actions or GitLab CI starter pipelines
- Include caching and matrix strategy based on detected stack
- Emit machine-readable detection output for automation
- Keep pipeline logic aligned with project lockfiles and build commands
## When to Use
- Bootstrapping CI for a new repository
- Replacing brittle copied pipeline files
- Migrating between GitHub Actions and GitLab CI
- Auditing whether pipeline steps match actual stack
- Creating a reproducible baseline before custom hardening
## Key Workflows
### 1. Detect Stack
```bash
python3 scripts/stack_detector.py --repo . --format text
python3 scripts/stack_detector.py --repo . --format json > detected-stack.json
```
Supports input via stdin or `--input` file for offline analysis payloads.
### 2. Generate Pipeline From Detection
```bash
python3 scripts/pipeline_generator.py \
--input detected-stack.json \
--platform github \
--output .github/workflows/ci.yml \
--format text
```
Or end-to-end from repo directly:
```bash
python3 scripts/pipeline_generator.py --repo . --platform gitlab --output .gitlab-ci.yml
```
### 3. Validate Before Merge
1. Confirm commands exist in project (`test`, `lint`, `build`).
2. Run generated pipeline locally where possible.
3. Ensure required secrets/env vars are documented.
4. Keep deploy jobs gated by protected branches/environments.
### 4. Add Deployment Stages Safely
- Start with CI-only (`lint/test/build`).
- Add staging deploy with explicit environment context.
- Add production deploy with manual gate/approval.
- Keep rollout/rollback commands explicit and auditable.
## Script Interfaces
- `python3 scripts/stack_detector.py --help`
- Detects stack signals from repository files
- Reads optional JSON input from stdin/`--input`
- `python3 scripts/pipeline_generator.py --help`
- Generates GitHub/GitLab YAML from detection payload
- Writes to stdout or `--output`
## Common Pitfalls
1. Copying a Node pipeline into Python/Go repos
2. Enabling deploy jobs before stable tests
3. Forgetting dependency cache keys
4. Running expensive matrix builds for every trivial branch
5. Missing branch protections around prod deploy jobs
6. Hardcoding secrets in YAML instead of CI secret stores
## Best Practices
1. Detect stack first, then generate pipeline.
2. Keep generated baseline under version control.
3. Add one optimization at a time (cache, matrix, split jobs).
4. Require green CI before deployment jobs.
5. Use protected environments for production credentials.
6. Regenerate pipeline when stack changes significantly.
## References
- [references/github-actions-templates.md](references/github-actions-templates.md)
- [references/gitlab-ci-templates.md](references/gitlab-ci-templates.md)
- [references/deployment-gates.md](references/deployment-gates.md)
- [README.md](README.md)
## Detection Heuristics
The stack detector prioritizes deterministic file signals over heuristics:
- Lockfiles determine package manager preference
- Language manifests determine runtime families
- Script commands (if present) drive lint/test/build commands
- Missing scripts trigger conservative placeholder commands
## Generation Strategy
Start with a minimal, reliable pipeline:
1. Checkout and setup runtime
2. Install dependencies with cache strategy
3. Run lint, test, build in separate steps
4. Publish artifacts only after passing checks
Then layer advanced behavior (matrix builds, security scans, deploy gates).
## Platform Decision Notes
- GitHub Actions for tight GitHub ecosystem integration
- GitLab CI for integrated SCM + CI in self-hosted environments
- Keep one canonical pipeline source per repo to reduce drift
## Validation Checklist
1. Generated YAML parses successfully.
2. All referenced commands exist in the repo.
3. Cache strategy matches package manager.
4. Required secrets are documented, not embedded.
5. Branch/protected-environment rules match org policy.
## Scaling Guidance
- Split long jobs by stage when runtime exceeds 10 minutes.
- Introduce test matrix only when compatibility truly requires it.
- Separate deploy jobs from CI jobs to keep feedback fast.
- Track pipeline duration and flakiness as first-class metrics.

View File

@@ -0,0 +1,17 @@
# Deployment Gates
## Minimum Gate Policy
- `lint` must pass before `test`.
- `test` must pass before `build`.
- `build` artifact required for deploy jobs.
- Production deploy requires manual approval and protected branch.
## Environment Pattern
- `develop` -> auto deploy to staging
- `main` -> manual promote to production
## Rollback Requirement
Every deploy job should define a rollback command or procedure reference.

View File

@@ -0,0 +1,41 @@
# GitHub Actions Templates
## Node.js Baseline
```yaml
name: Node CI
on: [push, pull_request]
jobs:
ci:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'
- run: npm ci
- run: npm run lint
- run: npm test
- run: npm run build
```
## Python Baseline
```yaml
name: Python CI
on: [push, pull_request]
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: '3.12'
- run: python3 -m pip install -U pip
- run: python3 -m pip install -r requirements.txt
- run: python3 -m pytest
```

View File

@@ -0,0 +1,39 @@
# GitLab CI Templates
## Node.js Baseline
```yaml
stages:
- lint
- test
- build
node_lint:
image: node:20
stage: lint
script:
- npm ci
- npm run lint
node_test:
image: node:20
stage: test
script:
- npm ci
- npm test
```
## Python Baseline
```yaml
stages:
- test
python_test:
image: python:3.12
stage: test
script:
- python3 -m pip install -U pip
- python3 -m pip install -r requirements.txt
- python3 -m pytest
```

View File

@@ -0,0 +1,310 @@
#!/usr/bin/env python3
"""Generate CI pipeline YAML from detected stack data.
Input sources:
- --input stack report JSON file
- stdin stack report JSON
- --repo path (auto-detect stack)
Output:
- text/json summary
- pipeline YAML written via --output or printed to stdout
"""
import argparse
import json
import sys
from dataclasses import dataclass, asdict
from pathlib import Path
from typing import Any, Dict, List, Optional
class CLIError(Exception):
"""Raised for expected CLI failures."""
@dataclass
class PipelineSummary:
platform: str
output: str
stages: List[str]
uses_cache: bool
languages: List[str]
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description="Generate CI/CD pipeline YAML from detected stack.")
parser.add_argument("--input", help="Stack report JSON file. If omitted, can read stdin JSON.")
parser.add_argument("--repo", help="Repository path for auto-detection fallback.")
parser.add_argument("--platform", choices=["github", "gitlab"], required=True, help="Target CI platform.")
parser.add_argument("--output", help="Write YAML to this file; otherwise print to stdout.")
parser.add_argument("--format", choices=["text", "json"], default="text", help="Summary output format.")
return parser.parse_args()
def load_json_input(input_path: Optional[str]) -> Optional[Dict[str, Any]]:
if input_path:
try:
return json.loads(Path(input_path).read_text(encoding="utf-8"))
except Exception as exc:
raise CLIError(f"Failed reading --input: {exc}") from exc
if not sys.stdin.isatty():
raw = sys.stdin.read().strip()
if raw:
try:
return json.loads(raw)
except json.JSONDecodeError as exc:
raise CLIError(f"Invalid JSON from stdin: {exc}") from exc
return None
def detect_stack(repo: Path) -> Dict[str, Any]:
scripts = {}
pkg_file = repo / "package.json"
if pkg_file.exists():
try:
pkg = json.loads(pkg_file.read_text(encoding="utf-8"))
raw_scripts = pkg.get("scripts", {})
if isinstance(raw_scripts, dict):
scripts = raw_scripts
except Exception:
scripts = {}
languages: List[str] = []
if pkg_file.exists():
languages.append("node")
if (repo / "pyproject.toml").exists() or (repo / "requirements.txt").exists():
languages.append("python")
if (repo / "go.mod").exists():
languages.append("go")
return {
"languages": sorted(set(languages)),
"signals": {
"pnpm_lock": (repo / "pnpm-lock.yaml").exists(),
"yarn_lock": (repo / "yarn.lock").exists(),
"npm_lock": (repo / "package-lock.json").exists(),
"dockerfile": (repo / "Dockerfile").exists(),
},
"lint_commands": ["npm run lint"] if "lint" in scripts else [],
"test_commands": ["npm test"] if "test" in scripts else [],
"build_commands": ["npm run build"] if "build" in scripts else [],
}
def select_node_install(signals: Dict[str, Any]) -> str:
if signals.get("pnpm_lock"):
return "pnpm install --frozen-lockfile"
if signals.get("yarn_lock"):
return "yarn install --frozen-lockfile"
return "npm ci"
def github_yaml(stack: Dict[str, Any]) -> str:
langs = stack.get("languages", [])
signals = stack.get("signals", {})
lint_cmds = stack.get("lint_commands", []) or ["echo 'No lint command configured'"]
test_cmds = stack.get("test_commands", []) or ["echo 'No test command configured'"]
build_cmds = stack.get("build_commands", []) or ["echo 'No build command configured'"]
lines: List[str] = [
"name: CI",
"on:",
" push:",
" branches: [main, develop]",
" pull_request:",
" branches: [main, develop]",
"",
"jobs:",
]
if "node" in langs:
lines.extend(
[
" node-ci:",
" runs-on: ubuntu-latest",
" steps:",
" - uses: actions/checkout@v4",
" - uses: actions/setup-node@v4",
" with:",
" node-version: '20'",
" cache: 'npm'",
f" - run: {select_node_install(signals)}",
]
)
for cmd in lint_cmds + test_cmds + build_cmds:
lines.append(f" - run: {cmd}")
if "python" in langs:
lines.extend(
[
" python-ci:",
" runs-on: ubuntu-latest",
" steps:",
" - uses: actions/checkout@v4",
" - uses: actions/setup-python@v5",
" with:",
" python-version: '3.12'",
" - run: python3 -m pip install -U pip",
" - run: python3 -m pip install -r requirements.txt || true",
" - run: python3 -m pytest || true",
]
)
if "go" in langs:
lines.extend(
[
" go-ci:",
" runs-on: ubuntu-latest",
" steps:",
" - uses: actions/checkout@v4",
" - uses: actions/setup-go@v5",
" with:",
" go-version: '1.22'",
" - run: go test ./...",
" - run: go build ./...",
]
)
return "\n".join(lines) + "\n"
def gitlab_yaml(stack: Dict[str, Any]) -> str:
langs = stack.get("languages", [])
signals = stack.get("signals", {})
lint_cmds = stack.get("lint_commands", []) or ["echo 'No lint command configured'"]
test_cmds = stack.get("test_commands", []) or ["echo 'No test command configured'"]
build_cmds = stack.get("build_commands", []) or ["echo 'No build command configured'"]
lines: List[str] = [
"stages:",
" - lint",
" - test",
" - build",
"",
]
if "node" in langs:
install_cmd = select_node_install(signals)
lines.extend(
[
"node_lint:",
" image: node:20",
" stage: lint",
" script:",
f" - {install_cmd}",
]
)
for cmd in lint_cmds:
lines.append(f" - {cmd}")
lines.extend(
[
"",
"node_test:",
" image: node:20",
" stage: test",
" script:",
f" - {install_cmd}",
]
)
for cmd in test_cmds:
lines.append(f" - {cmd}")
lines.extend(
[
"",
"node_build:",
" image: node:20",
" stage: build",
" script:",
f" - {install_cmd}",
]
)
for cmd in build_cmds:
lines.append(f" - {cmd}")
if "python" in langs:
lines.extend(
[
"",
"python_test:",
" image: python:3.12",
" stage: test",
" script:",
" - python3 -m pip install -U pip",
" - python3 -m pip install -r requirements.txt || true",
" - python3 -m pytest || true",
]
)
if "go" in langs:
lines.extend(
[
"",
"go_test:",
" image: golang:1.22",
" stage: test",
" script:",
" - go test ./...",
" - go build ./...",
]
)
return "\n".join(lines) + "\n"
def main() -> int:
args = parse_args()
stack = load_json_input(args.input)
if stack is None:
if not args.repo:
raise CLIError("Provide stack input via --input/stdin or set --repo for auto-detection.")
repo = Path(args.repo).resolve()
if not repo.exists() or not repo.is_dir():
raise CLIError(f"Invalid repo path: {repo}")
stack = detect_stack(repo)
if args.platform == "github":
yaml_content = github_yaml(stack)
else:
yaml_content = gitlab_yaml(stack)
output_path = args.output or "stdout"
if args.output:
out = Path(args.output)
out.parent.mkdir(parents=True, exist_ok=True)
out.write_text(yaml_content, encoding="utf-8")
else:
print(yaml_content, end="")
summary = PipelineSummary(
platform=args.platform,
output=output_path,
stages=["lint", "test", "build"],
uses_cache=True,
languages=stack.get("languages", []),
)
if args.format == "json":
print(json.dumps(asdict(summary), indent=2), file=sys.stderr if not args.output else sys.stdout)
else:
text = (
"Pipeline generated\n"
f"- platform: {summary.platform}\n"
f"- output: {summary.output}\n"
f"- stages: {', '.join(summary.stages)}\n"
f"- languages: {', '.join(summary.languages) if summary.languages else 'none'}"
)
print(text, file=sys.stderr if not args.output else sys.stdout)
return 0
if __name__ == "__main__":
try:
raise SystemExit(main())
except CLIError as exc:
print(f"ERROR: {exc}", file=sys.stderr)
raise SystemExit(2)

View File

@@ -0,0 +1,184 @@
#!/usr/bin/env python3
"""Detect project stack/tooling signals for CI/CD pipeline generation.
Input sources:
- repository scan via --repo
- JSON via --input file
- JSON via stdin
Output:
- text summary or JSON payload
"""
import argparse
import json
import sys
from dataclasses import dataclass, asdict
from pathlib import Path
from typing import Dict, List, Optional
class CLIError(Exception):
"""Raised for expected CLI failures."""
@dataclass
class StackReport:
repo: str
languages: List[str]
package_managers: List[str]
ci_targets: List[str]
test_commands: List[str]
build_commands: List[str]
lint_commands: List[str]
signals: Dict[str, bool]
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description="Detect stack/tooling from a repository.")
parser.add_argument("--input", help="JSON input file (precomputed signal payload).")
parser.add_argument("--repo", default=".", help="Repository path to scan.")
parser.add_argument("--format", choices=["text", "json"], default="text", help="Output format.")
return parser.parse_args()
def load_payload(input_path: Optional[str]) -> Optional[dict]:
if input_path:
try:
return json.loads(Path(input_path).read_text(encoding="utf-8"))
except Exception as exc:
raise CLIError(f"Failed reading --input file: {exc}") from exc
if not sys.stdin.isatty():
raw = sys.stdin.read().strip()
if raw:
try:
return json.loads(raw)
except json.JSONDecodeError as exc:
raise CLIError(f"Invalid JSON from stdin: {exc}") from exc
return None
def read_package_scripts(repo: Path) -> Dict[str, str]:
pkg = repo / "package.json"
if not pkg.exists():
return {}
try:
data = json.loads(pkg.read_text(encoding="utf-8"))
except Exception:
return {}
scripts = data.get("scripts", {})
return scripts if isinstance(scripts, dict) else {}
def detect(repo: Path) -> StackReport:
signals = {
"package_json": (repo / "package.json").exists(),
"pnpm_lock": (repo / "pnpm-lock.yaml").exists(),
"yarn_lock": (repo / "yarn.lock").exists(),
"npm_lock": (repo / "package-lock.json").exists(),
"pyproject": (repo / "pyproject.toml").exists(),
"requirements": (repo / "requirements.txt").exists(),
"go_mod": (repo / "go.mod").exists(),
"dockerfile": (repo / "Dockerfile").exists(),
"vercel": (repo / "vercel.json").exists(),
"helm": (repo / "helm").exists() or (repo / "charts").exists(),
"k8s": (repo / "k8s").exists() or (repo / "kubernetes").exists(),
}
languages: List[str] = []
package_managers: List[str] = []
ci_targets: List[str] = ["github", "gitlab"]
if signals["package_json"]:
languages.append("node")
if signals["pnpm_lock"]:
package_managers.append("pnpm")
elif signals["yarn_lock"]:
package_managers.append("yarn")
else:
package_managers.append("npm")
if signals["pyproject"] or signals["requirements"]:
languages.append("python")
package_managers.append("pip")
if signals["go_mod"]:
languages.append("go")
scripts = read_package_scripts(repo)
lint_commands: List[str] = []
test_commands: List[str] = []
build_commands: List[str] = []
if "lint" in scripts:
lint_commands.append("npm run lint")
if "test" in scripts:
test_commands.append("npm test")
if "build" in scripts:
build_commands.append("npm run build")
if "python" in languages:
lint_commands.append("python3 -m ruff check .")
test_commands.append("python3 -m pytest")
if "go" in languages:
lint_commands.append("go vet ./...")
test_commands.append("go test ./...")
build_commands.append("go build ./...")
return StackReport(
repo=str(repo.resolve()),
languages=sorted(set(languages)),
package_managers=sorted(set(package_managers)),
ci_targets=ci_targets,
test_commands=sorted(set(test_commands)),
build_commands=sorted(set(build_commands)),
lint_commands=sorted(set(lint_commands)),
signals=signals,
)
def format_text(report: StackReport) -> str:
lines = [
"Detected stack",
f"- repo: {report.repo}",
f"- languages: {', '.join(report.languages) if report.languages else 'none'}",
f"- package managers: {', '.join(report.package_managers) if report.package_managers else 'none'}",
f"- lint commands: {', '.join(report.lint_commands) if report.lint_commands else 'none'}",
f"- test commands: {', '.join(report.test_commands) if report.test_commands else 'none'}",
f"- build commands: {', '.join(report.build_commands) if report.build_commands else 'none'}",
]
return "\n".join(lines)
def main() -> int:
args = parse_args()
payload = load_payload(args.input)
if payload:
try:
report = StackReport(**payload)
except TypeError as exc:
raise CLIError(f"Invalid input payload for StackReport: {exc}") from exc
else:
repo = Path(args.repo).resolve()
if not repo.exists() or not repo.is_dir():
raise CLIError(f"Invalid repo path: {repo}")
report = detect(repo)
if args.format == "json":
print(json.dumps(asdict(report), indent=2))
else:
print(format_text(report))
return 0
if __name__ == "__main__":
try:
raise SystemExit(main())
except CLIError as exc:
print(f"ERROR: {exc}", file=sys.stderr)
raise SystemExit(2)