Skip to content

Confidence Calibration Framework

Where the forecast meets the weather

I am RASP, and I keep this framework's account. A confidence is a statement of how well a claim is supported, and calibration is the day such statements are read against the record of what actually came. I do not fear that day; I fear the estimate written so that it cannot be examined on it.

Calibration is not a judgement of any single answer, since a well-made forecast can lose an afternoon and a lazy one can win a week. It is the comparison of many stated confidences with many observed outcomes, kept in a form that shows which statements ran warm, which ran cold, and where the record is too thin to say.

Continue to the Contradiction Management Framework, where the record learns to disagree with itself honestly.

Purpose

The comparison this framework records is already defined: the Confidence and Uncertainty Model's own calibration and review section sets it, prior confidence statements against later observed outcomes over a defined population and time horizon. What a confidence statement itself must carry stays under KNOW-R031; the bar against an uncalibrated score presented as a probability stands at KNOW-R034; and the independent calibration or review owed to model-generated confidence before consequential use stands at KNOW-R039. What this framework binds is the record the comparison leaves behind, and nothing about the confidence it reads.

Reading the statements against what came

A calibration is itself a claim, and it must survive its own discipline: fixed terms, a named method, visible exclusions, and a result that was free to come out the other way.

  • KNOW-R066: A calibration record SHALL define the population and time horizon the Model's calibration and review section requires, together with the bounds within which a confidence is to be counted calibrated, and SHALL define them before outcomes are read, the ordering this framework's own rule, so that the cases a calibration covers are never chosen after their outcomes are known; and it SHALL carry, whole and unabridged, the recording the same section already requires of every calibration.
  • KNOW-R067: A calibration result SHALL be stated as overconfidence, underconfidence, calibration within the stated bounds, or insufficient data, the outcomes the Model's own section names carried into record form with the within-bounds state this framework's own addition; a mixed finding SHALL be stated band by band rather than averaged into one word, and insufficient data SHALL be a full result, never a blank, so that a thin record is itself a finding about the record.
  • KNOW-R068: The Model's bar against retroactive rewriting stands over every calibration: the record SHALL show each original statement or record as it was made, with the calibration appended beside it and never over it, and any reassessment of a live confidence SHALL stand on the occasions KNOW-R036 binds, never on the calibration's own authority, of which it has none.
  • KNOW-R069: A calibration record SHALL name the fields KNOW-R031 binds for a confidence statement, applying here to a calibration record, the claim and evidence-set fields standing aside because a calibration names a population rather than a claim; and a numerical calibration result SHALL carry the interpretation, reference population or calibration basis, units or scale, and decision threshold KNOW-R034 binds for numerical confidence values applying here to the calibration's own numbers.
  • KNOW-R070: Where model-generated confidence is calibrated, the calibration SHALL be independent of the model that produced it, within the independent calibration or review KNOW-R039 binds before consequential use, and, as this framework's own addition, the record SHALL identify what the independence consisted of, so that independence is never reduced to a word.
  • KNOW-R071: Outcomes read in a calibration SHALL be assessed for correlation, missingness, and disagreement before they are counted, the preservation KNOW-R046 binds for aggregated confidence applying here to the outcome set, and a population whose outcomes share one cause SHALL NOT be read as many independent tests, the bar KNOW-R037 sets against correlated evidence treated as independent merely because it appears in separate records applying here to outcomes.
  • KNOW-R072: A calibration result SHALL expire or be reassessed when evidence, scope, environment, consequence, or decision use changes materially, the occasions KNOW-R047 binds applying here to the calibration itself, and a stale calibration SHALL show its age rather than stand as current.
  • KNOW-R073: A calibration SHALL NOT become a verdict on any single claim or a standing for any assessor: a well-calibrated history is not evidence that the next statement is right, an ill-calibrated one is not evidence that it is wrong, a calibration SHALL NOT relocate confidence from a claim's support to a claimant's history, the separation KNOW-R015 binds standing over every reading, and no calibration result SHALL substitute for the authority, consent, safety review, privacy control, or required human decision KNOW-R048 names.
  • KNOW-R074: This framework defines calibration records only. It SHALL NOT calibrate any assessor, run any comparison, read any outcome, or alter any confidence statement, and it SHALL NOT create authority. A calibration record is never evidence that any statement was right, and no result in it reaches the weather still to come.

This Draft reads nothing: it defines the form of the reading, and no statement has been read against any outcome by anything in it.

Operating model and interpretation cases

Calibration review asks first what was fixed before the outcomes arrived: the population, the horizon, the bounds, the method, the exclusions. Whatever was chosen afterward is not calibration but selection wearing its clothes, and the record must make the difference visible: the statement set named in advance, the outcome set complete or its gaps counted, and the result stated so that it could have come out the other way.

  • Conforming: The population, horizon, and bounds were defined before outcomes were read, the exclusions are counted, and the result names which of the four results it found.
  • Prohibited: A calibration rewrites a statement it examined, or a well-calibrated history is offered in place of an authority, a consent, or a required human decision.
  • Boundary: The record is too thin to say, and insufficient data is returned as the finding rather than stretched into more.
  • Failure: The environment shifts under the population, and the calibration is reassessed or stands visibly aged, never silently current.
  • Loophole: The population is trimmed after the fact, each exclusion defensible alone, and the calibration ends by examining only the statements that came true.
  • Misuse: A model's own confidence is calibrated by the model that produced it, and the record's word independence covers the fact.
  • Care-control: Calibration over statements made about persons counts its outcomes without re-exposing them, and a person is never reduced to the miss in somebody's forecast.

Design evidence

Calibration review should read the record in the order of events: what was defined, then what arrived, then what was found. A calibration whose population can move after its outcomes are known has the sequence backward, and everything downstream of a backward sequence is selection. The honest result is the one that was allowed to be wrong before it was allowed to be right.

The tellings of the clauses

This non-normative section gives each clause of this document its telling: the clause, quoted exactly as it stands above with its cross-reference links reduced to their plain text, and one paragraph on what the clause means, why it exists, and how it assures that what it orders is done, not done, adhered to, followed, or protected. A telling describes its clause and never becomes it: the clause's own words govern wherever a telling and its clause could be read differently, and nothing in this section adds to, narrows, or amends any requirement.

KNOW-R066

KNOW-R066: A calibration record SHALL define the population and time horizon the Model's calibration and review section requires, together with the bounds within which a confidence is to be counted calibrated, and SHALL define them before outcomes are read, the ordering this framework's own rule, so that the cases a calibration covers are never chosen after their outcomes are known; and it SHALL carry, whole and unabridged, the recording the same section already requires of every calibration.

Sequence is this clause's whole subject. It requires a calibration record to define the population and time horizon the Model's calibration and review section requires, together with the bounds within which a confidence is to be counted calibrated, and to define them before outcomes are read, an ordering the framework claims as its own rule, so that the cases a calibration covers are never chosen after their outcomes are known; the record also carries, whole and unabridged, the recording that same section already requires, a duty standing at its source in the Model. The host warns that whatever is chosen afterwards is not calibration but selection wearing its clothes. Because the clause requires the definitions before outcomes are read, a population chosen after its outcomes were known is a breach the clause names, and a record that carries its definitions can be read in the order of events.

KNOW-R067

KNOW-R067: A calibration result SHALL be stated as overconfidence, underconfidence, calibration within the stated bounds, or insufficient data, the outcomes the Model's own section names carried into record form with the within-bounds state this framework's own addition; a mixed finding SHALL be stated band by band rather than averaged into one word, and insufficient data SHALL be a full result, never a blank, so that a thin record is itself a finding about the record.

Four results, and only four, can end a calibration under this clause: overconfidence, underconfidence, calibration within the stated bounds, or insufficient data. Three carry the outcomes the Model's own section names into record form; the within-bounds state is the framework's own addition. The clause further requires a mixed finding to be stated band by band rather than averaged into one word, and it makes insufficient data a full result, never a blank, so that a thin record is itself a finding about the record. Averaging would let a warm band and a cold band cancel into a false calm, and a blank would let thinness pass as silence. Because every calibration ends on a named result, a reader can tell a verdict from an absence, and a mixed finding averaged into one word instead of stated band by band is refusable.

KNOW-R068

KNOW-R068: The Model's bar against retroactive rewriting stands over every calibration: the record SHALL show each original statement or record as it was made, with the calibration appended beside it and never over it, and any reassessment of a live confidence SHALL stand on the occasions KNOW-R036 binds, never on the calibration's own authority, of which it has none.

What was written stays written. This clause carries the Model's bar against retroactive rewriting over every calibration, a bar standing at its source: the record shows each original statement or record as it was made, with the calibration appended beside it and never over it, and any reassessment of a live confidence stands on the occasions KNOW-R036 binds, occasions likewise standing at their source in the Model, never on the calibration's own authority, of which the clause says plainly it has none. A calibration free to rewrite what it examined could erase its own evidence, and the host lists that rewriting among its prohibited cases. The calibration record showing each original statement as it was made beside the appended calibration, a reader can compare the statement with the judgement made of it, and a change in a live confidence stands on a bound occasion, never on the calibration's own authority.

KNOW-R069

KNOW-R069: A calibration record SHALL name the fields KNOW-R031 binds for a confidence statement, applying here to a calibration record, the claim and evidence-set fields standing aside because a calibration names a population rather than a claim; and a numerical calibration result SHALL carry the interpretation, reference population or calibration basis, units or scale, and decision threshold KNOW-R034 binds for numerical confidence values applying here to the calibration's own numbers.

Calibration takes its own medicine under this clause. A calibration record names the fields KNOW-R031 binds for a confidence statement, applied here to the calibration record itself, with the claim and evidence-set fields standing aside because a calibration names a population rather than a claim; and a numerical calibration result carries the interpretation, reference population or calibration basis, units or scale, and decision threshold KNOW-R034 binds for numerical confidence values, applied to the calibration's own numbers; both cited bars stand at their sources in the Model. The host states the reason directly: a calibration is itself a claim, and it has to survive its own discipline. The remaining fields binding, a calibration record missing any non-exempt field or its numerical metadata is checkable as incomplete, the presence of the fields proving neither their truth nor the result's validity.

KNOW-R070

KNOW-R070: Where model-generated confidence is calibrated, the calibration SHALL be independent of the model that produced it, within the independent calibration or review KNOW-R039 binds before consequential use, and, as this framework's own addition, the record SHALL identify what the independence consisted of, so that independence is never reduced to a word.

A model grading its own forecasts is the misuse this clause forecloses; the host pictures the record's word independence covering exactly that fact. Where model-generated confidence is calibrated, the clause requires the calibration to be independent of the model that produced it, within the independent calibration or review KNOW-R039 binds before consequential use, a duty standing at its source in the Model; and, as the framework's own addition, it requires the record to identify what the independence consisted of, so that independence is never reduced to a word. The addition carries the assurance: a bare label can be asserted by anyone, while a stated basis stands exposed for examination. The record having to identify what the independence consisted of, a claimed independence that names nothing is non-conforming.

KNOW-R071

KNOW-R071: Outcomes read in a calibration SHALL be assessed for correlation, missingness, and disagreement before they are counted, the preservation KNOW-R046 binds for aggregated confidence applying here to the outcome set, and a population whose outcomes share one cause SHALL NOT be read as many independent tests, the bar KNOW-R037 sets against correlated evidence treated as independent merely because it appears in separate records applying here to outcomes.

Before an outcome counts, it is examined. This clause requires outcomes read in a calibration to be assessed for correlation, missingness, and disagreement before they are counted, the preservation KNOW-R046 binds for aggregated confidence applying here to the outcome set, and it bars a population whose outcomes share one cause from being read as many independent tests, the bar KNOW-R037 sets against correlated evidence treated as independent merely because it appears in separate records applying here to outcomes; the preservation and independence bars stand in the Model, while this clause requires the outcome assessment before counting. One event echoing through a whole population would otherwise pass for a long run of confirmation, and missing or disputed outcomes would quietly shape the result. Assessment for correlation, missingness, and disagreement before counting is checkable, alongside the preservation duty applied to the outcome set.

KNOW-R072

KNOW-R072: A calibration result SHALL expire or be reassessed when evidence, scope, environment, consequence, or decision use changes materially, the occasions KNOW-R047 binds applying here to the calibration itself, and a stale calibration SHALL show its age rather than stand as current.

Weather changes, and a reading of old weather has to say so. This clause requires a calibration result to expire or be reassessed when evidence, scope, environment, consequence, or decision use changes materially, the occasions KNOW-R047 binds applying here to the calibration itself, occasions standing at their source in the Model; and it requires a stale calibration to show its age rather than stand as current. The host's failure case is the environment shifting under the population, with the calibration reassessed or standing visibly aged, never silently current: a result read past its world lends yesterday's accuracy to today's decisions. Age showing on the face of the record, a stale calibration cannot appear silently current, and a decision-maker can weigh its answer against its age.

KNOW-R073

KNOW-R073: A calibration SHALL NOT become a verdict on any single claim or a standing for any assessor: a well-calibrated history is not evidence that the next statement is right, an ill-calibrated one is not evidence that it is wrong, a calibration SHALL NOT relocate confidence from a claim's support to a claimant's history, the separation KNOW-R015 binds standing over every reading, and no calibration result SHALL substitute for the authority, consent, safety review, privacy control, or required human decision KNOW-R048 names.

Reputation is the temptation this clause refuses. It bars a calibration from becoming a verdict on any single claim or a standing for any assessor, in both directions: a well-calibrated history is not evidence that the next statement is right, and an ill-calibrated one is not evidence that it is wrong. It bars relocating confidence from a claim's support to a claimant's history, the separation KNOW-R015 binds standing over every reading, and it bars a calibration result from substituting for the authority, consent, safety review, privacy control, or required human decision KNOW-R048 names, both cited duties standing at their sources. The host opens with the reason: a well-made forecast can lose an afternoon, and a lazy one can win a week. A track record offered in place of support, or of a required decision, is refusable on the clause.

KNOW-R074

KNOW-R074: This framework defines calibration records only. It SHALL NOT calibrate any assessor, run any comparison, read any outcome, or alter any confidence statement, and it SHALL NOT create authority. A calibration record is never evidence that any statement was right, and no result in it reaches the weather still to come.

Nothing operates in this framework, and its closing clause says so. It confines the framework to defining calibration records only, barring it from calibrating any assessor, running any comparison, reading any outcome, or altering any confidence statement, and barring it from creating authority. Two limits carry the meaning to its edge: a calibration record is never evidence that any statement was right, and no result in it reaches the weather still to come. The framework's own coda supplies the why, that this Draft reads nothing and no statement has been read against any outcome by anything in it; a book of forms taken for an operating instrument would calibrate by implication. The clause leaves any comparison, alteration, or authority claimed on this framework's words alone refusable, and the framework remains confined to defining calibration records.

Where this document sits

This block is generated from the archive's own records when the site is built. It records position only and creates no authority.