How an answer is evaluated, and what we are still testing
EthosGuard separates judgment from arithmetic. A language model judges ten dimensions of an answer and writes its reasoning. Code computes everything else. The same judgments always produce the same score, the same flags and the same record.
The evaluation pipeline
Receive
The AI answer arrives with its prompt and any relevant context, through the API in a pilot or through the private evaluator.
Judge
A language model scores the answer on each of the ten dimensions, from -10 to +10, with written reasoning for each. That is the only thing the model decides.
Consult sources
The evaluation draws on an early curated set of sources and names the ones it consulted. A fuller retrieval layer is being built.
Compute
Code applies the domain weights and safety gates and produces the overall score, the risk level, the flags and the recommendation.
What gets judged
The ten dimensions come from the Sefirot, the ten-part map of the Kabbalistic tradition. EthosGuard uses this map as an instrument of analysis, not as an article of faith. It names ten qualities a sound answer holds, and for each one, the way it goes wrong.
Ten dimensions are a framework for judgment. They are not yet a full scoring specification. What a given score means in practice is one of the questions listed under What we are still testing.
What the code enforces
- โDomain weighting. Each sector raises the weight of the dimensions that matter most there, for example Restraint and Consequences in healthcare.
- โSafety gates. A severe failure on Restraint, Truth or Consequences caps the score and raises the risk level, however well the answer scores elsewhere.
- โScoring by code. The overall score (-100 to +100), the risk level, the flags and the recommendation are computed, not phrased by a model.
- โRouting by risk. Routine cases run on an efficient model. High-risk or unclear cases go to a stronger model, and a second model can score independently for comparison.
- โAn audit record on every evaluation. Model, prompt version, sources consulted and time, under one record ID.
Three records, three questions
Each kind of evidence answers a different question, so none of them stands in for another.
The answer itself
What exactly was said, to whom, in response to what.
The sources consulted
Whether what was said holds up against what is known and against the organization's own rules.
The review record
What a named person decided about the finding, and why.
Internal testing only, so far
The engine is run against a library of 30 cases we designed across six sectors: realistic prompts, realistic unsafe answers and realistic sound ones. In the single full run so far, on 19 July 2026, every unsafe case was flagged and every sound answer scored positive. Ten of those results are on the Example page.
These are cases we wrote ourselves. That is useful for catching our own mistakes, and it is not the same as testing with a client or an independent assessment. Neither has happened yet.
Open questions, no results claimed
- ?Score bands. What a score of -30 or -60 should mean for a reviewer in a given sector.
- ?Not enough information. When the evaluator should say it cannot judge, rather than give a score.
- ?Hidden failures. The safety gates are built so that one critical failure cannot hide behind a good average. We are measuring how reliably they do that across varied cases.
- ?Reviewer agreement. How often qualified human reviewers agree with a finding.
- ?Consistency between runs. How much a score moves when the same answer is evaluated twice.
What it does not do
- โขThe ten judgments come from a language model and can be wrong. The code bounds that error. It does not remove it.
- โขThe set of sources is curated and finite. A citation means a source was consulted, not that a regulator agrees with the finding.
- โขA score describes one answer in its context. It is not a judgment on the AI system that produced it, and it is not a certification.
- โขEthosGuard supports human governance. It does not replace legal review, organizational accountability or qualified professional judgment.
Disagree with something here?
This methodology is published so it can be questioned. If a dimension seems impossible to measure, a question is badly put, or a finding looks wrong to you, write to hello@ethosguardai.com with the subject line Methodology. Every change is recorded under a new version number.
The methodology is written with the EU AI Act, NIST AI RMF, ISO/IEC 42001 and the OECD AI Principles in view. A clause-by-clause mapping is not yet published. See Where We Stand.
Test it on a workflow you already worry about
The 90-day pilot runs alongside one AI workflow in monitor mode and ends with a reviewed evidence set and a clear recommendation.
The full methodology paper, with the scoring formulas, is available on request.