CloudAuditEnv — LLM Agent Evaluation for Cloud Security

Procedurally generated cloud security audit environment that tests LLM agents on finding and fixing real vulnerability patterns — built for the OpenENV MetaHackathon.

PythonFastAPIPydanticOpenAI SDKhttpxOpenEnvQwen2.5-72B-InstructDockerHuggingFace SpacesUvicorn

Repository →

The Problem

LLM agents are increasingly deployed in infrastructure management roles, but there's a shortage of environments that test whether they can actually perform security operations — not just identify them. I built CloudAuditEnv to generate randomized AWS-like environments with real vulnerability patterns (public SSH access, unencrypted storage, wildcard IAM policies) and measure whether an agent can remediate them without breaking essential services in the process. I built this for the OpenENV MetaHackathon.

What I Built

Each episode randomly generates 13–22 AWS-style resources with a mix of vulnerabilities. The agent observes the full environment state and outputs flat JSON actions to remediate issues. The interesting design problem I ran into was the health score: an agent that blindly closes every open port breaks production. So I built remediation as constrained optimization rather than greedy fix-everything — removing an essential security group rule costs -0.2, breaking IAM access costs -0.3, high enough to matter without making any single mistake fatal.

LLM Agent (Qwen2.5-72B) ──HTTP/JSON──▶ FastAPI Server (port 7860)
                                              │
                                    CloudAuditEnv
                                       ├── Procedural Generator (uuid + random)
                                       ├── State Manager (SG/S3/RDS/EBS/IAM)
                                       └── Reward + Health Score Computer

Key Decisions & Tradeoffs

I used a flat, verbose action model over discriminated unions — every action carries some unused fields, but parsing works reliably across Pydantic versions instead of breaking on edge cases.
I clamped scores to (0.1, 0.9) — this avoids validation failures at the boundaries, at the cost of perfect remediation never showing as a clean 1.0.
I didn't add seed support — each episode is genuinely random, which means agents can't memorize resource IDs, but it also means there's no reproducible run-to-run comparison yet.
I added a batch remediation action (remediate_all_in_sg) after discovering that fixing security group rules one at a time exhausted the 30-step budget on complex groups.

Highlights

Procedurally generates 13–22 randomized AWS-style resources with real vulnerability patterns (public SSH, unencrypted storage, wildcard IAM)
Health score system penalizes blind fixes — removing essential security group rules costs -0.2, breaking IAM costs -0.3
Baseline Qwen2.5-72B agent scores ~0.44 on hardest IAM remediation — wildcard permission removal as genuinely hard constrained optimization
Flat JSON action model with batch remediation (remediate_all_in_sg) to stay within 30-step budget
Built for OpenENV MetaHackathon with honest documentation of limitations (no seed support, unverified benchmark scores)

The Problem

What I Built

LLM Agent (Qwen2.5-72B) ──HTTP/JSON──▶ FastAPI Server (port 7860) │ CloudAuditEnv ├── Procedural Generator (uuid + random) ├── State Manager (SG/S3/RDS/EBS/IAM) └── Reward + Health Score Computer

Key Decisions & Tradeoffs

I used a flat, verbose action model over discriminated unions — every action carries some unused fields, but parsing works reliably across Pydantic versions instead of breaking on edge cases.

I clamped scores to (0.1, 0.9) — this avoids validation failures at the boundaries, at the cost of perfect remediation never showing as a clean 1.0.

I didn't add seed support — each episode is genuinely random, which means agents can't memorize resource IDs, but it also means there's no reproducible run-to-run comparison yet.

I added a batch remediation action (remediate_all_in_sg) after discovering that fixing security group rules one at a time exhausted the 30-step budget on complex groups.

Highlights

Procedurally generates 13–22 randomized AWS-style resources with real vulnerability patterns (public SSH, unencrypted storage, wildcard IAM)

Health score system penalizes blind fixes — removing essential security group rules costs -0.2, breaking IAM costs -0.3

Baseline Qwen2.5-72B agent scores ~0.44 on hardest IAM remediation — wildcard permission removal as genuinely hard constrained optimization

Flat JSON action model with batch remediation (remediate_all_in_sg) to stay within 30-step budget

Built for OpenENV MetaHackathon with honest documentation of limitations (no seed support, unverified benchmark scores)