Improving Labeling Consistency with Detailed Constitutional Definitions and AI-Driven Evaluation

Enterprises need to know exactly what their systems detect, and that definition must stay consistent over time. Writing a definition precise enough to settle every hard case has long been impractical because human annotators cannot hold a document that detailed in working memory. In our research paper, Single-Source Safety Definitions, we replace the human interpreter with AI and show that LLMs can hold, apply, and maintain specifications far longer and more precise than any annotator can, making the definition itself the single source of truth for classification, labeling, retraining, and customer-facing explanations. For our Cisco AI Defense product portfolio, we are moving our full safety taxonomy to this AI-first model. We also extend this approach beyond safety classifications, as shown in Defining Model Provenance: A Constitution for AI Supply Chain Safety and Security.

Cisco’s Integrated AI Security and Safety Framework organizes the threats enterprises face when…