Skip to main content
SkyDynamics aeroEBT instructor app on iPad: flight phase panel, My Notepad, live event and competency observable-behaviour marks during an EBT session.
Flight deck of the SkyDynamics A320 FNPT II simulator

EBT practice guide

Instructor concordance. Measuring and closing the grading gap.

Concordance is how closely two instructors observing the same crew reach the same assessment. Unmeasured, it decays quietly and your competency data becomes partly noise. This guide covers how to measure it and run a calibration cycle that holds.

Contextual Composition Protocol

Concordance starts before the session. Every instructor works from the same reference.

Most concordance work happens after the fact: measure the drift, then calibrate. The Contextual Composition Protocol, the SkyDynamics technology behind every aeroEBT application, also works upstream. When an instructor records and assesses, the instructor app already holds the scenario phase, the event or failure in play, the observable behaviours it is designed to elicit, and your competency framework and grading scale. Every instructor starts from the same reference, so there is less drift to correct.

  • The scenario, the event or failure in play and its expected behaviours held behind every observation
  • Each record correlated with the exact moment of the session it belongs to
  • Every grade carries its justification: the moment, the event and the expected behaviours it was assessed against
  • The protocol never intervenes, and no instructor has to learn how to use it
  • Drift still measured against the fleet baseline, so the effect is evidenced rather than assumed
ScenarioSETINSERTOBSERVEINSERTPublishPhase · ClimbSAWWLMOBSERVESAW12345OfflineSame scenarioAI-assisted · instructor in control
Every observation correlated with the moment it belongs to.Illustrative

Definition

Three words, three meanings. Concordance, inter-rater reliability and the referent rater.

They get used interchangeably, and should not be. The chart shows a fourth idea, severity and leniency: where one instructor's grades sit against the fleet. It is a different measure from inter-rater reliability.

  • Concordance

    Do two instructors observing the same performance agree on what they saw and how they assessed it?

  • Inter-rater reliability

    The statistical expression of that agreement. It needs several instructors grading the same performance, which is what a referent session provides.

  • Referent rater

    A reference assessment of a recorded session that instructors are compared against during calibration.

aeroEBT shows each instructor's grade distribution against your fleet baseline, from the grades instructors already enter.

Concordance in aeroEBT

Causes

Why grading drifts apart. Six causes, none of them an instructor failing.

They are programme design problems that show up in the data.

  • The scale means different things to different people

    Without worked examples, "average" is an instructor's personal midpoint rather than a shared standard.

  • Observation is harder than grading

    Seeing the behaviour, and deciding which competency it shows, takes more skill than choosing the number. The grade is the easy part.

  • Severity and leniency are stable traits

    Individual instructors tend to be consistently harsher or softer than the group. That is measurable and correctable.

  • Procedural competencies absorb everything else

    A low grade for procedures often really reflects workload management or decision-making. The number hides the cause.

  • Calibration is annual, drift is continuous

    A yearly standardisation day cannot hold a standard that moves every month.

  • Nobody sees the aggregate

    Instructors see their own sessions. Without a fleet view, no one can see that two bases have separated.

aeroEBT shows grading drift per instructor against the fleet baseline, so your standards team can target calibration where it matters instead of re-briefing everyone.

How aeroEBT keeps a programme from drifting

Method

Measure concordance honestly. Two measures in six steps, and the last step sends you round again.

Two different measures answer two different questions. The distribution analysis shows who grades higher or lower than the fleet. Inter-rater reliability shows whether instructors agree on one performance, and it needs a referent session.

Measured, then re-measured

  • Normalise the data

    Compare like with like: same competency, comparable scenario phase, comparable crew experience.

  • Establish the baseline

    The fleet distribution per competency is the reference, not an abstract ideal.

  • Position each instructor

    The distribution analysis: each instructor's grades against that baseline, including how often they use the extremes of the scale. It shows severity and leniency, not agreement.

  • Add a referent check

    Inter-rater reliability: a recorded session graded by everyone gives the direct comparison that live sessions cannot.

  • Read the narrative

    Where grades agree but notes diverge, the observation is drifting even though the numbers look healthy.

  • Re-measure after calibration

    A calibration session that does not move the distribution did not work. Measure it.

In aeroEBT the distribution analysis comes from the grades instructors already enter, with no extra step in the session. The referent session is the one separate exercise, once a quarter.

See what it asks of your instructors

Operating rhythm

A calibration cycle that holds. Four rhythms across the year.

  • Monthly

    Training standards reviews the distribution per instructor and per base. Outliers are a conversation, not a sanction.

  • Quarterly

    A referent-rater session: everyone grades the same recorded session, then discusses the differences rather than the grades.

  • Twice a year

    Worked examples are refreshed so the scale is anchored to current operations, not last decade's.

  • Annually

    Instructor standardisation is evidenced for the authority with the data behind it, not a statement that it happened.

In aeroEBT calibration sessions are planned and recorded in the same system as the training, and measured for effect.

Calibrate in the ORCA cycle

In aeroEBT

Concordance in the workflow. Not a separate project.

aeroEBT shows each instructor's grade distribution against the fleet baseline, from the grades your instructors already enter, and keeps calibration sessions and their outcomes in the same record as the training itself.

  • Distribution per instructor, per competency, per base
  • Calibration sessions planned, recorded and measured for effect
  • Observations extracted from instructor notes so narrative drift is visible too
  • Evidence for your authority produced from the same data
Before12345CalibrationAfter · re-measured12345Fleet baselineInstructorMeasured, not steered
A calibration session, measured for effect.Illustrative

Questions training standards managers ask

It is the programme an operator runs to keep instructor assessments consistent in EBT: instructors are trained to a shared standard, their grading is measured against each other and against the fleet, and calibration closes the gaps that appear. The IATA EBT Implementation Guide calls it the Instructor Concordance Assurance Program (ICAP), and an approved EBT programme is expected to have one. The six-step method and the calibration cycle on this page are a working ICAP.

aeroEBT, from SkyDynamics, measures each instructor's grade distribution against the fleet baseline from the grades already entered in the iPad instructor app, so there is no separate form. Calibration sessions are planned and recorded in the same system and the comparison is repeated afterwards, which gives the authority the evidence of an ICAP. The same platform covers EBT scenario design, CBTA grading and training management, and it is in use at Olympic Air and AnimaWings.

Not necessarily. Severity is a trait, not an error, and a strict instructor may be observing more carefully. What matters is that the difference is known, discussed and stable rather than invisible.

Enough that an instructor's distribution is not dominated by a handful of crews. In practice that means reading trends over a training cycle rather than reacting to single sessions.

It should not. The purpose is a shared standard. Programmes that publish league tables get defensive grading, which is worse than drift.

Much grading variation starts before the grade: instructors look for different things because they carry different context in their heads. The Contextual Composition Protocol, SkyDynamics technology running behind every aeroEBT application, holds that context in memory instead. When an instructor records and assesses in the ORCA cycle, the instructor app already knows the scenario phase, the event or failure in play, the observable behaviours it is designed to elicit and the competency framework, and it correlates what the instructor records with that moment. Observations start from a shared reference, and measurement and calibration then close the gap that remains.

No. Assessment of human performance stays with the instructor. AI is useful for extracting observations from notes and showing patterns across a population that no individual can see.

Evidence that standardisation is managed: how it is measured, what was found, what was done about it and whether it worked.

See where your instructors actually stand.

In a 30-minute demo we look at how your grading is distributed today and what a calibration cycle would change.

We respond within one business day.