AI SOC Platform RFP Checklist for Security Teams and MSSPs

A vendor-neutral checklist for evaluating security AI in production.

Reviewed by ContraForce team

An AI SOC platform RFP should test production behavior, not presentation quality. Require each vendor to demonstrate evidence handling, tenant isolation, action authority, failure behavior, and a reproducible operating outcome using the same scenarios.

1. Scope and definitions

• Define whether the product is an assistant, AI SOC analyst, automation platform, agent platform, MDR service, or delivery platform.
• List every workflow stage the product supports: triage, investigation, response, ticketing, reporting, and tuning.
• Define what “autonomous,” “resolved,” “closed,” and “response time” mean.
• Identify which capabilities are generally available, preview, roadmap, or partner-delivered.

2. Evidence and investigation

• Which data sources can the agent query?
• Does the final verdict include the evidence used?
• Can an operator see failed queries, missing evidence, and contradictory observations?
• How does the system avoid unsupported conclusions?
• Can investigation requirements differ by incident class and customer?

3. Actions and human control

• Separate read, recommend, approve, and execute permissions.
• List every supported response action by integration.
• Demonstrate an approval gate for a consequential action.
• Explain how confidence thresholds relate to action authority.
• Show what happens when an approver does not respond.
• Preserve the identity and time of every approval.

4. Multitenancy

• Demonstrate two customer tenants with different procedures and permissions.
• Explain how credentials, prompts, retrieved context, and evidence remain isolated.
• Show customer-specific service boards, SLAs, and reports.
• Describe how shared procedures receive customer exceptions without uncontrolled forks.
• Explain offboarding and access revocation.

5. Integrations

• Identify whether each integration is read-only, bidirectional, or action-capable.
• Document required roles and delegated access.
• State API-rate, data-volume, and event-delay constraints.
• Show retry, deduplication, and dead-letter behavior.
• Identify who maintains the connector when an upstream API changes.

6. Security and data architecture

• Where are prompts, evidence, logs, credentials, and model inputs stored?
• Is customer telemetry copied into a vendor-controlled data lake?
• Which model providers process data and under what retention terms?
• How are secrets stored and rotated?
• What audit certifications and penetration-test evidence are available?
• Can the platform support regional or customer-specific data requirements?

7. Model governance

• Which decisions use deterministic logic versus a model?
• How are model and prompt versions tracked?
• Can a customer pin or review a model change?
• How are evaluations constructed and regression-tested?
• What happens when the model is unavailable?
• Are outputs used for model training, and can customers opt out?

8. Service workflow

• Does the product create a customer-ready ticket or only an investigation summary?
• Can it preserve internal versus external notes?
• Does it update reports and service metrics?
• Can it feed incident classification into detection tuning?
• How are exceptions assigned and escalated?

9. Commercial model

• Identify platform, workspace, user, data, model, and per-incident charges.
• Define what counts as a billable incident or run.
• Model cost under quiet, typical, and surge volumes.
• State minimum terms, overages, implementation fees, and exit costs.
• Explain whether customers can export their procedures and evidence.

10. Performance claims

• Require the event boundaries, population, measurement window, exclusions, and statistic type for every claim.
• Distinguish vendor telemetry, customer-reported outcomes, independent testing, and modeled estimates.
• Ask for median and percentile values, not only a mean or best case.
• Reproduce key claims during the proof of value.

Review the ContraForce benchmark methodology as an example of the definitions and limitations ContraForce applies to its own claims.

Required proof-of-value scenarios

1. Routine benign incident with sufficient evidence
2. Confirmed threat requiring a human approval
3. Ambiguous incident with missing evidence
4. Upstream API or credential failure
5. Duplicate or updated incident event
6. Two tenants with different action authority
7. Ticket synchronization and customer-ready closure

Score the observable outcome, analyst touches, evidence quality, safety, and recovery behavior. Do not award points for roadmap promises.