National Security Innovation Network: NSIN Presents: SynGen Challenge

NSIN, in partnership with the National Geospatial-Intelligence Agency (NGA), the Defense Intelligence Agency (DIA), and the Defense Innovation Unit (DIU) seeks solutions for the generation of synthetic training data for computer vision (CV) algorithms. Open to all eligible innovators.

Deadline Extension

The application deadline has been extended to June 1, 2023 at 11:59 p.m. ET.

Overview

The National Security Innovation Network (NSIN), in partnership with the National Geospatial-Intelligence Agency (NGA), Defense Intelligence Agency (DIA), and Defense Innovation Unit (DIU), are running a prize challenge to identify research and solutions to support a critical need for the generation of synthetic training data for computer vision (CV) algorithms.

White paper submissions will be evaluated by a team of subject matter experts (SMEs) with backgrounds in imagery, computer vision, model creation, GIS, GeoINT, systems engineering, and cloud computing. After an initial evaluation, up to the top 10 companies will be selected as finalists to participate in the pitch event and submission of final deliverables, which were used in the research for and generation of their white paper. The top performing company will be awarded $75,000 in prize money.

NGA and DIA are making significant investments in this arena and looking to investigate the efficacy and availability of synthetic training data for CV algorithms in mid-early stage ventures that can support various defense applications.

Benefits of Participating

$75,000 award to the top finalist company.
Exposure to leaders across multiple branches of the U.S. Department of Defense (DoD) and elements of the United States Intelligence Community responsible for the development and deployment of synthetic training data technologies.
Opportunities to learn from and collaborate with other industry leaders and partners in this space.
Potential for follow-on opportunities such as Other Transaction Agreements (OTAs), Cooperative Research and Development Agreements (CRADAs), and others.

Apply Now

To participate in the Challenge, interested participants need to create an account on IdeaScale. Upon registration you will receive a verification email. Check your spam folder if the verification email is not received. Please contact challenges@nsin.mil for questions about IdeaScale registration. All relevant information regarding this Challenge can be found via the following link to the IdeaScale platform:

Apply Now on IdeaScale

Eligibility

U.S. Owned and Operated Companies (to include those with non-U.S. citizens)
Mid- to Early-Stage Ventures OR non-traditional defense contractors with the ability to demonstrate the solution’s capabilities by June 2023
Registration in SAM.gov (Not a requirement for initial submission to the challenge. Failure to register in SAM.gov before the Round 2 submission date (see schedule below) will result in elimination from the challenge)

Why Participate

As the amount of imagery data collected increases, there is a growing need for a way to quickly process and interpret that imagery. Computer vision models will be required to do this in a timely manner and these models will require a large amount of high-quality, organized, well-labeled data. Synthetically generated training data may be able to help with this, but there is a lack of quality, repeatable data science on the subject, especially in regards to overhead data.

Many papers, projects, and challenges focus on tweaking computer vision models to achieve optimal results. With this challenge, NSIN/NGA/DIA/DIU are looking to collect data on the efficacy of synthetics and how making changes to the way synthetic data, used in training computer vision models, will affect the quality of those computer vision models. This information will allow the U.S. Government to better target areas of research, better allocate compute and human resources, and provide rationale for those decisions.

Timeline

Dec. 9, 2022: Solicitation Release
Dec. 16, 2022: Ask Me Anything Session
June 1, 2023: Solicitation Closes & Down-selection Begins
June 12, 2023: Down-selection Ends & Finalists Announced
June 15, 2023: Pitch Rehearsal
June 22, 2023: Virtual Pitch Event
Week of June 26, 2023: Winner Announced

Judging Criteria

Proposal Round

The quality of the white paper in the initial submission will be evaluated against five major criteria:

Well-written: clearness & conciseness
Content: Addresses key questions
Standards: Within parameters and meets standards
Evidence: Contains scientific language & evidence-based solutions
Adaptability: Ability to use this approach for other use cases

Pitch Round

Those invited to the pitch event will submit the materials developed and used in the research for and generation of the white paper and will be evaluated against five major criteria:

Novelty: Innovation of proposed solution or technique.
Commercialization: Company has a solution that is viable in non-defense markets.
Quality of 3D Model Files: Representation of target of interest, speaks clearly to questions.
Code Quality: Written clearly & concisely, efficient, well-commented.
Image Chips/Annotation Quality: Correctly formatted, speaks to key questions.

Problem Statement

The purpose of the challenge is to answer critical questions about the generation and efficacy of synthetic training data for computer vision (CV) algorithms. The submissions have a set of requirements that must be met. The problem statement defines areas that must be addressed, the technical and deliverable requirements are listed below and define the parameters and expectations. NGA/DIA/DIU will use the results of this challenge to determine areas for future research and resource utilization, including potential funding.

NGA aims to gather research and answers concerning the following three questions:

How does the inclusion of shadows and varying the sun angle affect the optimization of the CV network?
How does the fidelity of the 3D model affect the optimization of the CV network?
How does the resolution of the synthetic imagery data affect the optimization of the CV network?

The subject of this challenge is exterior industrial air conditioning units. These are common items that should be reasonably detectable using CV algorithms. In the real world, these air conditioning units come in all sizes and shapes, with various support features such as piping and ducting. The participants will create 3D models and Blender Scene files.

Materials Provided

In the following days, the following materials will be provided to participants via the official challenge website:

Three sets of corner coordinates representing example Areas of Interest (AOIs). These are only example areas. Participants may use any imagery and AOIs they feel are appropriate.
A Dockerfile with an untrained computer vision model for the participants to train. As an example, a model such as Mask RCNN Answers to any questions asked, subject to discretion. All questions and answers will be provided to all participants. By submitting a question, you are agreeing to this release.
Example image chips with appropriate annotation.

**NOTE: Compute resources, software, and imagery will not be provided.

Technical Requirements

In order to ensure proper evaluation and replication of participant results, participants will be expected to:

Generate synthetic scenes using the software Blender, version 3.0 or higher. (Free and open-source, Blender is available at: https://www.blender.org/). -Refrain from providing workflows which require closed-source or commercial software, including any software which may require training fees for use (such as DIRSIG). Participants may use closed-source and commercial software during the challenge, but such software cannot be required to evaluate or reproduce the results. (As examples, participants may use Microsoft Windows or macOS as a base for running Blender, use Adobe software to create reports, etc. Examples of an impermissible workflow would be requiring AutoCAD to load or process 3D models, Maya to render image chips, etc.)

Deliverables

Participants will deliver the following by 11:59 p.m. ET on the close of the solicitation period, May 1, 2023 an IEEE-formatted whitepaper (not to exceed 20 pages), in PDF format, to include at least the following information (more information may be included if desired):

Abstract
System Information (OS/Hardware/Software)
For each of the three questions: Statement of Question, Methodology, System Information (if different from that listed at the top of the paper)
Model Weights and Parameters Used
Model Performance (Include things like render and training time taken, etc. Also include information such as false positive/negative rates, etc.)
Summary (Recommended courses of action related to question)
Overall summary

Pitch Event

Participants selected as finalists will pitch their solutions in a virtual pitch event the week of June 19, 2023. The pitch will include five minutes for a public presentation (to all registered attendees) and 10 minutes for a private questions and answers session from the judging panel.

A formal process for the submission of deliverables will be provided between the start of the competition and the date of the solicitation closing. Additional information will be provided on the solicitation page on IdeaScale. Finalists selected to participate in the pitch event will be evaluated and one (1) winner will be selected to receive $75,000 in prize money.

Follow-On Activities

This NSIN Challenges Open Call Announcement is considered to have potential for further efforts that may be accomplished via FAR-based contracting instruments, Other Transaction Authority (OTA) for Prototype Projects 10 USC 4022 and Research 10 USC 4021, Prizes for advanced technology achievements 10 USC 4025, and/or Prize Competitions 15 USC 3719. If a prototype OTA is awarded and considered successfully completed, follow-on production may be pursued in accordance with 10 USC 4022(f). The public open call announcement made on the NSIN website is considered to satisfy the reasonable effort to obtain competition in accordance with 10 USC 4025(b), 15 USC 3719(e) and 10 USC 4022(b)(2). Any FAR-based actions will follow announcement procedures per FAR 5.201(b) accordingly actions will follow announcement procedures per FAR 5.201(b) accordingly.

Intellectual Property Considerations

By participating in this competition and submitting in any of the rounds, participants are granting the US Government and its Partners permanent access to the provided white paper with the provision that the U.S. Government and its Partners may store and copy the white paper in perpetuity. Participants will retain intellectual ownership of and publishing rights to the white paper.

Excluding white papers and any material considered to be co-produced with the U.S. Government, the U.S. Government and its Partners agree not to share submissions with other entities, including files and tradecraft concepts.

Background Information

The 2025 NGA Strategy calls for action to deliver trusted GEOINT where and when needed to meet mission needs today, while simultaneously looking to the future to ensure we maintain comparative advantage over competitors and adversaries. Navy Vice Admiral Robert D. Sharp, former NGA Director, released the strategy with specified goals. Goal 4 focuses on translating actionable intelligence into opportunities to ensure our national security and reduce strategic surprise by elevating our understanding of the world and anticipating future events. In support of the NGA Strategy and goal 4, NGA is conducting this challenge to answer critical questions about the generation and efficacy of synthetic training data. “We face significant changes in the domestic and global environment: we must be ready to meet the 21st century challenges and to recognize emerging threats and opportunities.” Daniel R. Coats, Director of National Intelligence, National Intelligence Strategy of the United States of America 2019.

About National Geospatial-Intelligence Agency
The National Geospatial-Intelligence Agency (NGA) delivers world-class geospatial intelligence that provides a decisive advantage to policymakers, military service members, intelligence professionals and first responders. Anyone who sails a U.S. ship, flies a U.S. aircraft, makes national policy decisions, fights wars, locates targets, responds to natural disasters, or even navigates with a cellphone relies on NGA. NGA enables all of these critical actions and shapes decisions that impact our world through the indispensable discipline of geospatial intelligence (GEOINT). NGA receives guidance and oversight from DOD, the Director of National Intelligence (DNI) and Congress. NGA is headquartered in Springfield, Virginia, and has two major locations in St. Louis and Arnold, Missouri. Approximately 14,500 civilian, military and contractor employees work across more than 100 locations in the U.S. and 20 international locations. Hundreds of NGA employees also serve on support teams at U.S. military, diplomatic and allied locations around the world.

About Defense Intelligence Agency (DIA) Our diverse workforce is skilled in military history and doctrine, economics, physics, chemistry, world history, political science, bio-sciences and computer sciences. We travel the world, meeting and working closely with professionals from foreign countries. At DIA, we provide military intelligence to warfighters, defense policymakers and force planners in the Department of Defense and Intelligence Community, in support of U.S. military planning and operations and weapon systems acquisition. We plan, manage and execute intelligence operations during peacetime, crisis and war.

About Defense Innovation Unit (DIU) The Defense Innovation Unit (DIU) strengthens our national security by accelerating the adoption of commercial technology throughout the military and strengthening our allied and national security innovation bases. DIU partners with organizations across the Department of Defense (DoD), from the services and components to combatant commands and defense agencies, to rapidly prototype and field advanced commercial solutions that address national security challenges. With offices in Silicon Valley, Boston, Austin, Chicago and the Pentagon, DIU connects its DoD partners with leading technology companies across the country.

About NSIN The National Security Innovation Network is a program of the U.S. Department of Defense that collaborates with major universities and the venture community to develop solutions that drive national security innovation. We operate two portfolios of programs and services: Venture and Talent. Together, these portfolios form a pipeline of activities and solutions that accelerate the pace of defense innovation.