Nexus - AI Task Orchestrator

02-infrastructure-health
Procedimiento: Verificaci├│n de Salud de Infraestructura

Documentacion
1 plugin(s)
Titulo
Descripcion
Contenido Markdown
13656 caracteres
# Procedimiento: Verificaci├│n de Salud de Infraestructura

## Metadata
- **ID**: PROC-02
- **Frecuencia**: Diario (health checks), Semanal (SSL/certificados), Mensual (completo)
- **Duraci├│n estimada**: 30-60 min
- **Requiere**: Acceso a servidores, Docker, endpoints de salud
- **Dependencias**: Ninguna
- **Bloquea**: Deploy si hay certificados < 7 d├¡as para expirar

## Objetivo

Verificar proactivamente el estado de la infraestructura: certificados SSL, health endpoints, espacio en disco, conectividad a servicios externos, y recursos de contenedores.

## Checklist Ejecutable

### 1. Verificar Certificados SSL

```powershell
# PowerShell - Verificar certificado de un dominio
function Test-SSLCertificate {
    param([string]$Domain, [int]$Port = 443)
    
    $tcpClient = New-Object System.Net.Sockets.TcpClient
    $tcpClient.Connect($Domain, $Port)
    
    $sslStream = New-Object System.Net.Security.SslStream($tcpClient.GetStream(), $false, {$true})
    $sslStream.AuthenticateAsClient($Domain)
    
    $cert = $sslStream.RemoteCertificate
    $expiry = [DateTime]::Parse($cert.GetExpirationDateString())
    $daysLeft = ($expiry - (Get-Date)).Days
    
    $tcpClient.Close()
    
    [PSCustomObject]@{
        Domain = $Domain
        Expiry = $expiry
        DaysLeft = $daysLeft
        Status = if ($daysLeft -lt 7) { "CRITICAL" } elseif ($daysLeft -lt 30) { "WARNING" } else { "OK" }
    }
}

# Verificar dominios del proyecto
$domains = @(
    "api.ejemplo.com",
    "app.ejemplo.com"
)

$results = $domains | ForEach-Object { Test-SSLCertificate -Domain $_ }
$results | Format-Table -AutoSize

# Alertar si hay cr├¡ticos
$critical = $results | Where-Object { $_.Status -eq "CRITICAL" }
if ($critical) {
    Write-Host "ÔÜá´©Å CERTIFICADOS CR├ìTICOS:" -ForegroundColor Red
    $critical | Format-Table
}
```

```bash
# Linux/bash alternativa
check_ssl() {
    domain=$1
    expiry=$(echo | openssl s_client -servername $domain -connect $domain:443 2>/dev/null | openssl x509 -noout -enddate | cut -d= -f2)
    expiry_epoch=$(date -d "$expiry" +%s)
    now_epoch=$(date +%s)
    days_left=$(( ($expiry_epoch - $now_epoch) / 86400 ))
    
    if [ $days_left -lt 7 ]; then
        echo "CRITICAL: $domain expires in $days_left days ($expiry)"
    elif [ $days_left -lt 30 ]; then
        echo "WARNING: $domain expires in $days_left days ($expiry)"
    else
        echo "OK: $domain expires in $days_left days"
    fi
}

# Verificar
check_ssl "api.ejemplo.com"
check_ssl "app.ejemplo.com"
```

- [ ] Certificados verificados
- [ ] Ninguno expira en < 30 d├¡as
- [ ] Si hay WARNING/CRITICAL: ticket creado para renovaci├│n

### 2. Health Checks de Endpoints

```powershell
# PowerShell - Verificar endpoints de salud
function Test-HealthEndpoint {
    param(
        [string]$Name,
        [string]$Url,
        [int]$TimeoutSeconds = 10
    )
    
    try {
        $stopwatch = [System.Diagnostics.Stopwatch]::StartNew()
        $response = Invoke-WebRequest -Uri $Url -TimeoutSec $TimeoutSeconds -UseBasicParsing
        $stopwatch.Stop()
        
        [PSCustomObject]@{
            Name = $Name
            Url = $Url
            Status = $response.StatusCode
            ResponseTimeMs = $stopwatch.ElapsedMilliseconds
            Healthy = $response.StatusCode -eq 200
        }
    }
    catch {
        [PSCustomObject]@{
            Name = $Name
            Url = $Url
            Status = "ERROR"
            ResponseTimeMs = -1
            Healthy = $false
            Error = $_.Exception.Message
        }
    }
}

# Endpoints a verificar
$endpoints = @(
    @{ Name = "API Health"; Url = "http://localhost:5000/health" },
    @{ Name = "Workers Health"; Url = "http://localhost:5001/health" },
    @{ Name = "RabbitMQ Management"; Url = "http://localhost:15672/api/health/checks/alarms" }
)

$results = $endpoints | ForEach-Object { Test-HealthEndpoint -Name $_.Name -Url $_.Url }
$results | Format-Table -AutoSize

# Resumen
$healthy = ($results | Where-Object { $_.Healthy }).Count
$total = $results.Count
Write-Host "`nHealth: $healthy/$total endpoints OK" -ForegroundColor $(if ($healthy -eq $total) { "Green" } else { "Red" })
```

```bash
# Bash alternativa
check_health() {
    name=$1
    url=$2
    
    start=$(date +%s%N)
    status=$(curl -s -o /dev/null -w "%{http_code}" --max-time 10 "$url")
    end=$(date +%s%N)
    
    duration=$(( ($end - $start) / 1000000 ))
    
    if [ "$status" = "200" ]; then
        echo "Ô£à $name: OK (${duration}ms)"
    else
        echo "ÔØî $name: FAILED (status: $status)"
    fi
}

check_health "API" "http://localhost:5000/health"
check_health "Workers" "http://localhost:5001/health"
check_health "RabbitMQ" "http://localhost:15672/api/health/checks/alarms"
```

- [ ] Todos los endpoints responden 200
- [ ] Response time < 1000ms
- [ ] Si hay fallos: investigar inmediatamente

### 3. Espacio en Disco

```powershell
# PowerShell - Windows
Get-WmiObject Win32_LogicalDisk | 
    Where-Object { $_.DriveType -eq 3 } |
    Select-Object DeviceID, 
        @{N='SizeGB';E={[math]::Round($_.Size/1GB,2)}},
        @{N='FreeGB';E={[math]::Round($_.FreeSpace/1GB,2)}},
        @{N='UsedPercent';E={[math]::Round((($_.Size - $_.FreeSpace) / $_.Size) * 100, 1)}},
        @{N='Status';E={
            $used = (($_.Size - $_.FreeSpace) / $_.Size) * 100
            if ($used -gt 90) { "CRITICAL" }
            elseif ($used -gt 80) { "WARNING" }
            else { "OK" }
        }} |
    Format-Table -AutoSize
```

```bash
# Linux
df -h | awk 'NR==1 || /^\/dev/' | while read line; do
    if [ "$line" != "Filesystem" ]; then
        used=$(echo $line | awk '{print $5}' | tr -d '%')
        if [ $used -gt 90 ]; then
            echo "CRITICAL: $line"
        elif [ $used -gt 80 ]; then
            echo "WARNING: $line"
        else
            echo "OK: $line"
        fi
    fi
done
```

- [ ] Ning├║n disco > 80% usado
- [ ] Si > 80%: identificar qu├® consume espacio
- [ ] Si > 90%: acci├│n inmediata (limpiar logs, docker prune)

### 4. Estado de Contenedores Docker

```powershell
# Estado de contenedores
docker ps -a --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}" 

# Contenedores no healthy
docker ps -a --filter "health=unhealthy" --format "{{.Names}}: {{.Status}}"

# Contenedores reiniciando
docker ps -a --filter "status=restarting" --format "{{.Names}}"

# Uso de recursos
docker stats --no-stream --format "table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.MemPerc}}"
```

```bash
# Verificar contenedores cr├¡ticos est├ín running
REQUIRED_CONTAINERS="rabbitmq orchestrator-api orchestrator-workers"

for container in $REQUIRED_CONTAINERS; do
    status=$(docker inspect -f '{{.State.Status}}' $container 2>/dev/null)
    if [ "$status" = "running" ]; then
        echo "Ô£à $container: running"
    else
        echo "ÔØî $container: $status (or not found)"
    fi
done
```

- [ ] Todos los contenedores requeridos running
- [ ] Ninguno en estado unhealthy o restarting
- [ ] Memory usage < 80% del limit

### 5. Conectividad a Servicios Externos

```powershell
# Verificar conectividad a servicios externos
function Test-ServiceConnectivity {
    param(
        [string]$Name,
        [string]$Host,
        [int]$Port
    )
    
    try {
        $tcpClient = New-Object System.Net.Sockets.TcpClient
        $connect = $tcpClient.BeginConnect($Host, $Port, $null, $null)
        $wait = $connect.AsyncWaitHandle.WaitOne(5000, $false)
        
        if ($wait) {
            $tcpClient.EndConnect($connect)
            $tcpClient.Close()
            [PSCustomObject]@{ Name = $Name; Host = "$Host`:$Port"; Status = "OK" }
        } else {
            [PSCustomObject]@{ Name = $Name; Host = "$Host`:$Port"; Status = "TIMEOUT" }
        }
    }
    catch {
        [PSCustomObject]@{ Name = $Name; Host = "$Host`:$Port"; Status = "FAILED"; Error = $_.Exception.Message }
    }
}

# Servicios a verificar
$services = @(
    @{ Name = "RabbitMQ"; Host = "localhost"; Port = 5672 },
    @{ Name = "PostgreSQL"; Host = "localhost"; Port = 5432 },
    @{ Name = "SQL Server"; Host = "localhost"; Port = 1433 },
    @{ Name = "Redis"; Host = "localhost"; Port = 6379 }
)

$results = $services | ForEach-Object { Test-ServiceConnectivity -Name $_.Name -Host $_.Host -Port $_.Port }
$results | Format-Table -AutoSize
```

- [ ] Todos los servicios accesibles
- [ ] Si hay fallos: verificar firewalls, credenciales, estado del servicio

### 6. Verificar Queues de RabbitMQ

```powershell
# Usando RabbitMQ Management API
$rabbitUrl = "http://localhost:15672/api/queues"
$cred = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes("admin:admin"))

$queues = Invoke-RestMethod -Uri $rabbitUrl -Headers @{ Authorization = "Basic $cred" }

$queues | Select-Object name, messages, consumers, state | Format-Table -AutoSize

# Alertar si hay colas con muchos mensajes sin consumidores
$problematic = $queues | Where-Object { $_.messages -gt 1000 -and $_.consumers -eq 0 }
if ($problematic) {
    Write-Host "ÔÜá´©Å COLAS SIN CONSUMIDORES:" -ForegroundColor Yellow
    $problematic | Format-Table name, messages
}
```

- [ ] Todas las colas tienen consumidores
- [ ] No hay acumulaci├│n excesiva de mensajes (> 1000)

### 7. Generar Reporte de Salud

```powershell
# Generar reporte consolidado
$report = @{
    Timestamp = Get-Date -Format "yyyy-MM-dd HH:mm:ss"
    SSL = @{ Status = "OK"; Details = "All certificates valid > 30 days" }
    Endpoints = @{ Status = "OK"; Healthy = 3; Total = 3 }
    Disk = @{ Status = "OK"; MaxUsage = "65%" }
    Containers = @{ Status = "OK"; Running = 5; Total = 5 }
    Services = @{ Status = "OK"; Connected = 4; Total = 4 }
    RabbitMQ = @{ Status = "OK"; QueuedMessages = 0 }
}

# Calcular estado general
$overallStatus = if ($report.Values.Status -contains "CRITICAL") { "CRITICAL" }
                 elseif ($report.Values.Status -contains "WARNING") { "WARNING" }
                 else { "HEALTHY" }

Write-Host "`n========================================" -ForegroundColor Cyan
Write-Host "  INFRASTRUCTURE HEALTH REPORT" -ForegroundColor Cyan
Write-Host "  $($report.Timestamp)" -ForegroundColor Cyan
Write-Host "========================================" -ForegroundColor Cyan
Write-Host "`n  Overall Status: $overallStatus" -ForegroundColor $(
    switch ($overallStatus) {
        "HEALTHY" { "Green" }
        "WARNING" { "Yellow" }
        "CRITICAL" { "Red" }
    }
)
Write-Host ""
```

## Umbrales de Alerta

| M├®trica | OK | Warning | Critical |
|---------|----|---------| ---------|
| SSL Expiry | > 30 d├¡as | 7-30 d├¡as | < 7 d├¡as |
| Disk Usage | < 80% | 80-90% | > 90% |
| Endpoint Response | < 500ms | 500-2000ms | > 2000ms o timeout |
| Queue Messages | < 100 | 100-1000 | > 1000 sin consumidor |
| Container Memory | < 70% | 70-85% | > 85% |

## Resultado

- **├ëxito (HEALTHY)**:
  - Todos los certificados v├ílidos > 30 d├¡as
  - Todos los endpoints responden 200
  - Disco < 80% en todos los vol├║menes
  - Todos los contenedores running y healthy
  - Conectividad OK a todos los servicios

- **Warning**:
  - Certificado expira en < 30 d├¡as
  - Disco entre 80-90%
  - Alg├║n endpoint lento (> 1s)

- **Critical**:
  - Certificado expira en < 7 d├¡as ÔåÆ Renovar INMEDIATAMENTE
  - Disco > 90% ÔåÆ Limpiar espacio
  - Endpoint no responde ÔåÆ Investigar servicio
  - Contenedor unhealthy ÔåÆ Revisar logs, reiniciar

## Acciones de Remediaci├│n

| Problema | Acci├│n Inmediata | Acci├│n Permanente |
|----------|------------------|-------------------|
| SSL < 7 d├¡as | Renovar certificado manual | Configurar auto-renewal |
| Disco > 90% | `docker system prune`, limpiar logs | A├▒adir almacenamiento, retenci├│n de logs |
| Endpoint down | Reiniciar servicio | Investigar causa ra├¡z, a├▒adir redundancia |
| Container unhealthy | `docker restart` | Revisar health check, recursos |
| Queue acumulada | Verificar/reiniciar consumers | Escalar consumers, alertas |

## Mensaje de Finalizaci├│n

**ACCI├ôN REQUERIDA AL FINALIZAR:**

```bash
echo "====== PROCESO TERMINADO [$(date +%H%M%S)] ======" && echo "RESULTADO: Infrastructure Health - STATUS, SSL OK, Endpoints X/Y, Disk max Z%"
```


---

## Output Estructurado (Nexus)

Al finalizar, el agente DEBE generar un bloque JSON con el siguiente formato para que Nexus pueda procesarlo automaticamente:

```json
{
  "result": "success",
  "summary": "Ejecucion de PROC-02 completada. [Descripcion breve de resultados]",
  "metrics": {
    "issues_found": 0,
    "issues_resolved": 0,
    "ssl_days_left": 45, "endpoints_healthy": 5,
    "custom": {
      "procedure_specific_metric": "value"
    }
  },
  "backlog_items": [
    {
      "title": "Titulo del item de seguimiento",
      "description": "Descripcion detallada si se requiere accion futura",
      "priority": "medium",
      "type": "improvement",
      "tags": ["proc-02"]
    }
  ],
  "next_steps": [
    "Accion recomendada 1",
    "Accion recomendada 2"
  ],
  "warnings": [
    "Advertencias encontradas durante la ejecucion"
  ]
}
```

**Campos requeridos:**
- `result`: `"success"` | `"partial"` | `"failed"`
- `summary`: Resumen ejecutivo en 1-3 lineas

**Metricas especificas de este procedure:**
- ssl_days_left, endpoints_healthy, disk_max_percent, containers_healthy

**Criterios de resultado:**
- `success`: Procedimiento completado sin errores criticos
- `partial`: Completado con algunos problemas menores o items pendientes
- `failed`: Error critico o no se pudo completar

## Historial de Ejecuciones

| Fecha | Ejecutor | Status | SSL | Endpoints | Disk | Issues |
|-------|----------|--------|-----|-----------|------|--------|
| | | | | | | |