FRTE Face In Video Evaluation (FIVE) 2024 (original) (raw)
FRTE Face In Video Evaluation (FIVE) 2024
Participation Agreement | API and Test Plan | Validation | Encryption
Status
[2024-08-23]The FIVE Phase 2 submission deadline has been extended from 2024-08-30 to 2024-09-03.
[2024-07-01]Report cards have been distributed to developers that participated in Phase 1 of the test. The Phase 2 submission window runs from 2024-07-01 to 2024-08-30.
[2024-03-07]The final version of the test plan and API and validation package are now available.
[2024-02-20]An updated version of the draft test plan and API is now available. This document is available for public comment until 2024-02-29.
[2024-01-23]The Face in Video Evaluation (FIVE) 2024 activity is being conducted to assess the capability of face recognition algorithms to correctly identify or ignore persons appearing in video sequences. A draft test plan and API is now available for public comment until 2024-02-29.
Schedule
| 2024-03-07 | Final API published |
|---|---|
| 2024-03-11 | Phase 1 submission window opens (1 algorithm submission per developer) |
| 2024-05-18 | Phase 1 submission window closes |
| 2024-06 | Phase 1 report cards distributed |
| 2024-07-01 | Phase 2 submission window opens (Up to 2 algorithm submissions per developer) |
| 2024-09-03 | Phase 2 submission window closes |
| 2024-09 | Phase 2 report cards distributed |
| 2024-Q4/2025-Q1 | Public report published |
Note: Depending on the number of participants, the number of algorithm submissions and/or the number of submission phases may be revised or extended.
Motivation
The Face in Video Evaluation (FIVE) 2024 is being conducted to assess the capability of face recognition algorithms to correctly identify or ignore persons appearing in video sequences – i.e., the open-set identification problem. FIVE 2024 will include datasets and use cases involving degraded video imagery (low resolution, compressed, etc.) collected
- Outdoors with directional lighting
- At long range (300m+) and potentially affected by atmospheric turbulence
- From elevated platforms (large look-down pitch angles)
- With multiple people in the scene FIVE 2024 will also include datasets and use cases previously assessed in FIVE 2015, including
- High volume screening of persons in the crowded spaces (e.g. an airport)
- Low volume forensic examination of footage from a crime scene (e.g. a convenience store)
- Persons in business meetings (e.g. for video-conferencing)
- Persons appearing in television footage
Both comparative and absolute accuracy measures are of interest, given the goals to determine which algorithms are most effective and whether any are viable for the various use cases. Different applications differ in their tolerance of false positives, whether a human examiner will review outputs, the prior probabilities of mate vs. non-mate presence, and the cost of recognition error.
How to Participate
To participate in this evaluation, developers need to submit a participation agreement to NIST, wrap their software behind the published C++ API, run their libraries through the provided validation package (which creates a submission package), encrypt the package, and provide a download link for the encrypted submission package. More details are provided below.
Participation Agreement
FIVE 2024 is conducted by NIST, an agency of the United States Government. Participation is free of charge. FIVE 2024 is open to a global audience of face recognition developers. All organizations who seek to participate in FIVE 2024 must sign all pages of thisParticipation Agreement and submit it with their algorithm submission using the Submission Form. [last update: 2024-01-23]
[NOTE:] Organizations that have already submitted a participation agreement for FRTE Ongoing 1:1 do not need to send in a new participation agreement UNLESS the organization updates their cryptographic signing key.
API Document
A draft API and test plan has been published for public comment. After a final API has been published, all algorithms submitted must adhere to the published C++ API. [last update: 2024-01-23]
Validation
A validation package will be published by 2024-03-07. All participants must run their software through the validation package prior to submission. The purpose of validation is to ensure consistent algorithm output between your execution and NIST’s execution. [last update: 2024-01-23]
Encryption
All submissions must be properly encrypted and signed before transmission to NIST. This must be done according to theseinstructions using the FRTE Ongoing public key linked from thispage. Participants must email their public key to NIST. The participant’s public key must correspond to the participant’s public-key fingerprint provided on the signed Participation Application. [last update: 2024-01-23]
Submission
All algorithm submissions must be submitted through the Submission Form, which requires encrypted files be provided as a download link from a generic http server (e.g., Google Drive). We cannot accept Dropbox links. NIST will not register, or establish any kind of membership, on the provided website. Participants can submit their algorithm(s), participation agreement, and GPG key at the same time via the submission form. [last update: 2024-01-23]
Participants must subscribe to the FRTE/FATE mailing list to receive emails when new reports are published or announcements are made.
Contact Information
Inquiries and comments may be submitted to [email protected].Subscribe to the evaluation mailing list to receive emails when announcements or updates are made.
Acknowledgements
This research is based upon work supported in part by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA).