Face Analysis Technology Evaluation (FATE) Quality (original) (raw)

Status

[2025-05-18]A new version of the FATE SIDD Report is now available. It includes results for Secunet, Veridium, Vsoft, and Igd. It includes a new set for distributions of unified quality score on high quality images, for images exhibiting only minor departures from ISO/IEC 39794-5 Annex D photographic guidance.

[2025-04-03]A new version of the FATE SIDD Report is now available. It includes new results for Viante.

[2025-03-02]A new version of the FATE SIDD Report is now available. It includes new results for Secunet.

[2025-01-23]A new version of the FATE SIDD Report is now available. It includes new results for Secunet, Viante, and Cmcati.

[2025-12-08]A new version of the FATE SIDD Report is now available. It includes new results for Secunet, Ofiq1, and Veridium.

[2025-08-07]A new version of the FATE SIDD Report is now available. It includes new results for Daon, Innovatrics, and Pixelall.

[2025-06-17]A new version of the FATE SIDD Report is now available. It includes a new set for Distance from Eyes to Edges containing images with varying pose and background uniformity in which one eye is very close to the edge of the image, or partially out of the frame.

[2025-05-22]A new version of the FATE SIDD Report is now available. It includes new results for Mobbeel and Veridium and introduces a Resolution set with blur from air turbulence, instead of synthetic blur. The new plot is in section 3.14.3.

[2025-04-15]A new version of the FATE SIDD Report is now available. It includes additional plots showing the dependence of the Resolution quality measure on relative blur, the width of Gaussian blur normalized by inter-eye distance.

[2025-03-14]A new version of the FATE SIDD Report is now available. It includes results for Pixelall.

[2025-01-21]A new version of the FATE SIDD Report is now available. It includes results for IGD, Veridium, Mobbeel, ROC, and Kasikorn Labs.

We have also added a Kiosk dataset to the leaderboard below.

[2024-12-09]A new version of the FATE SIDD Report is now available. It includes results for Viante.

[2024-11-15]A new version of the FATE SIDD Report is now available. It introduces cumulative distribution plots by region-of-birth and sex, as well as demographic analysis for Unified Quality Score.

[2024-10-04]A new version of the FATE SIDD Report is now available. It includes results for PAPIL11.

[2024-09-06]A new version of the FATE SIDD Report is now available. It includes results for Ediffiqal and IGD. We have also added summary tables for Eyes Open 2 and Mouth Open 2.

[2024-08-08]A new version of the FATE SIDD Report is now available. It includes results for Mobbeel and Veridium.

[2024-07-03]A new version of the FATE SIDD Report is now available. It includes results for neurotechnology, qazsmartvisionai, and idemia.

We have added a section on demographics to address two topics: thresholds on quality measures, and demographic variability in quality measures.

Previous versions of the SIDD report can be found on our GitHub page.

[2024-05-13]A new version of the FATE SIDD Report is now available. It includes results for cu-face.

[2024-04-26]A new version of the FATE SIDD Report is now available. It includes results for two new submissions from FRP Kauai and Secunet.

The Quality Algorithm Performance leaderboard below has been updated with algorithms that implemented Unified Quality Score.

The Fate SIDD API Concept Document was updated on 4-23-2024 to be in accordance with ISO/IEC DIS 29794-5.

[2024-04-16]A new version of the FATE SIDD Report is now available. It includes results for two new submissions from IGD and ROC.

The Quality Algorithm Performance leaderboard below has been updated with algorithms that implemented Unified Quality Score.

Previous versions of the SIDD report can be found on our GitHub page.

[2024-04-05]A new version of the FATE SIDD API Concept Document is now posted. It sets a four-month period between submissions on slide 4 and specifies that the sign convention for pose estimates on slide 10 is the same as that in ISO/IEC 39794-5.

Previous versions of the API Concept Document can be found on our GitHub page.

[2024-03-29]A new version of the FATE SIDD Report is now available. It includes a correction to Pitch Set 2 results due to a systematic offset in the determination of ground truth for Pitch Set 2.

Previous versions of the SIDD report can be found on our GitHub page.

[2024-03-08]A new version of the FATE SIDD Report is now available. It includes results for one new submission from Mobbeel.

The Quality Algorithm Performance leaderboard below has been updated with algorithms that implemented Quality SIDD Unified Quality Score.

[2024-02-14]A new version of the FATE SIDD Report is now available. It includes results for two new submissions from ROC and Viante.

Previous versions of the SIDD report can be found on our GitHub page.

The Quality Algorithm Performance leaderboard below has been updated with algorithms that implemented Quality SIDD Unified Quality Score.

[2024-02-02]A new version of the FATE SIDD Report is now available. It includes results for a new submission from IGD.

Previous versions of the SIDD report can be found on our GitHub page.

[2023-12-29]A new version of the FATE SIDD Report is now available. It includes results for two new submissions from Datech and Vsoft.

Previous versions of the SIDD report can be found on our GitHub page.

[2023-12-15]A new version of the FATE SIDD Report is now available. It includes results for two new submissions from Seamfix and Neurotechnology. In addition, we have made a correction to the Unified Quality Score plot for Digidata, and introduced a set for inter-eye distance with non-zero yaw.

Previous versions of the SIDD report can be found on our GitHub page.

[2023-12-04]A new version of the FATE SIDD API Concept Document is now posted. It includes updated numbering for the ISO/IEC 29794-5 quality checks on slides 30 and 33, and current participation instructions on slide 4.

Previous versions of the API Concept Document can be found on our GitHub page.

[2023-12-01]A new version of the FATE SIDD Report is now available. It includes results for two new submissions from Secunet. The results for Eyes Open 2, Mouth Open 2, and Face Occlusion 2 measures are included. From this date forward, the Eyes Open, Mouth Open, and Face Occlusion measures are no longer being evaluated.

The quality leaderboard has been updated with algorithms that implemented Quality SIDD Unified Quality Score.

[2023-11-16]A new version of the FATE SIDD API Concept Document is now posted. We are planning to deprecate EyesOpen, MouthOpen, and FaceOcclusion and have introduced EyesOpen2, MouthOpen2, and FaceOcclusion2. The new measures are aligned with ISO/IEC 29794-5 CD3.

Developers may continue to implement EyesOpen, MouthOpen, and FaceOcclusion until November 30, 2023, after which those measures will no longer be evaluated. The new measures are now present in the API and can be implemented.

Previous versions of the API Concept Document can be found on our GitHub page.

[2023-11-06]A new version of the FATE SIDD Report is now available. It includes results for new submissions from October 2023. We have also changed median absolute error to mean absolute error and introduced two sets for manually determined pitch and yaw.

The quality leaderboard has been updated with algorithms that implemented Quality SIDD Unified Quality Score.

[2023-10-11]We are transitioning to evaluating Quality Summarization through only the “UnifiedQualityScore” component of Quality SIDD. Going forward, developers seeking to have their Quality Summarization algorithm evaluated should implement the Unified Quality Score component and submit to the Quality SIDD evaluation. The quality leaderboard has been updated to merge the results from the original Quality Summarization track with those from the Quality SIDD evaluation of Unified Quality Score.

[2023-09-20]We have released NISTIR 8485 - Face Analysis Technology Evaluation (FATE) Part 11: Face Image Quality Vector Assessment: Specific Image Defect Detection. As described in this press release, the report summarizes accuracy for seven submissions from five unique developers. We evaluated algorithms across 20 quality measures.
This report will be discussed at the November 2023 Face Image Quality Workshop.

[2023-08-18]A revision of the API and concept document for the Specific Image Defect Detection track has been postedhere.
We have updated references to FRVT to Face Analysis Technology Evaluation (FATE) and Face Recognition Technology Evaluation (FRTE).
Previous versions of the API document can be found on the GitHub page.

[2023-08-09]A revision of the API and concept document for the Specific Image Defect Detection track has been postedhere.
References to FRVT have been changed to FATE to reflect the use of face analysis, rather than face recognition.
Previous versions of the API document can be found on the GitHub page.

[2023-07-03]Starting today, all algorithms, participation agreements, and GPG keys should be submitted using the Submission Form.

How to Participate

To participate in this evaluation, developers need to submit a participation agreement to NIST, wrap their software behind the published C++ API, run their libraries through the provided validation package (which creates a submission package), encrypt the package, and provide a download link for the encrypted submission package. More details are provided below.

Participation Agreement

FATE is conducted by NIST, an agency of the United States Government. Participation is free of charge. FATE is open to a global audience of face recognition developers. All organizations who seek to participate in FATE must sign all pages of thisParticipation Agreement and submit it with their algorithm submission using the FRTE Submission Form. [last update: 2023-07-03]

[NOTE:] Organizations that have already submitted a participation agreement for FRTE Ongoing 1:1 do not need to send in a new participation agreement UNLESS the organization updates their cryptographic signing key.

API Document

A definitive API document has been published. All FATE APIs reference the supporting FRTE/FATE General Evaluation Specifications, which includes hardware and operating system environment, software requirements, reporting, and common data structures that support the APIs. All algorithms submitted must adhere to the published C++ API. [last update: 2023-04-06]

Validation

A validation package has been published. All participants must run their software through the updated validation package prior to submission. The purpose of validation is to ensure consistent algorithm output between your execution and NIST’s execution. [last update: 2023-04-28]

Encryption

All submissions must be properly encrypted and signed before transmission to NIST. This must be done according to theseinstructions using the FRTE/FATE Ongoing public key linked from thispage. Participants must email their public key to NIST. The participant’s public key must correspond to the participant’s public-key fingerprint provided on the signed Participation Application. [last update: 2019-04-23]

Submission

All algorithm submissions must be submitted through the Submission Form, which requires encrypted files be provided as a download link from a generic http server (e.g., Google Drive). We cannot accept Dropbox links. NIST will not register, or establish any kind of membership, on the provided website. Participants can submit their algorithm(s), participation agreement, and GPG key at the same time via the submission form. [last update: 2023-07-03]

Participants must subscribe to the FRVT mailing list to receive emails when new reports are published or announcements are made.

Subscribe to the FRVT mailing list to receive emails when announcements or updates are made.