Large Language Model (LLM) Evaluation Research Grant

Please review the program’s Terms & Conditions.

Two developers working together on a computer

Overview

Meta is pleased to invite university faculty to respond to this call for research proposals for LLM evaluations.

As AI models become increasingly complex and capable of responding to queries, and also acting on the world and reacting to the results of their actions, new benchmarks will be required to assess these advanced behaviors. These standards are critical to expand the scope of who the models can reach, how they interact with humans, and the types of tasks they can perform, while ensuring that these models can be trusted.

To foster further innovation in this area and deepen collaboration with academia, Meta is pleased to invite university teams to submit research proposals that contribute to building a new library of evaluations within three main areas: complex reasoning, emotional & social intelligence, and agentic behaviors.

A total of up to 4 awards of $200,000 are available each - and each award is inclusive of overhead. Funding will be provided to RFP winners pursuant to a Sponsored Research Agreement (SRA) containing open science terms. Please note that the terms of the SRA will not be subject to negotiation. We strongly encourage researchers from diverse backgrounds and of diverse abilities to apply.

The submission deadline is Friday September 6, at 11:59pm PST.

Research Grant Finalists

Large Language Model (LLM) Research Grant Finalists

In August 2024, we launched our first Research Grant specifically for academic researchers, looking for innovative proposals to build new benchmarks in the areas of general intelligence, emotional intelligence and agentic behaviors for large language models. We are now excited to announce the four finalists of this grant!

Areas of Interest

Meta encourages innovative proposals that will generate novel, challenging, ground truth benchmarks and evaluations, both for pre-training and post-training. We are specifically seeking benchmarks with human-generated data (we will disqualify synthetically generated data unless it has been disclosed and the proposal has a detailed explanation justifying the quality and usefulness of it).

Knowing that there are more than 7,000 known languages in the world - and LLMs are just beginning to be available and trusted in a fraction of them - we want to understand how we can scalably extend the coverage for capabilities we care about into many languages . How can we make sure models in different languages align with human values in a culturally sensitive way? How can we measure capabilities that we care about? Proposals that can scalably extend to cover multiple languages or modalities will be prioritized.

We encourage submissions that utilize evaluations in the following areas:

Complex Reasoning: As AI advances and becomes increasingly capable of interacting with humans and taking actions on their behalf, we need methods for understanding their reasoning and planning capabilities. How would we measure different types of intelligence and reasoning effectively as these models approach and/or surpass human performance? Proposals with a special focus on complex reasoning are encouraged, including a focus on complex mathematical reasoning, tool integration, hierarchical planning, memory and infinite context window, System 1 vs System 2 reasoning, multi-step deduction and induction, analogical thinking, and narrative analysis.
Emotional and Social Intelligence: If AI models and agent systems of the future are expected to interact with humans effectively, they will need not only cognitive intelligence but also emotional and social intelligence—the ability to understand human emotions and sentiments, respond emphatically, "read the room", and generally adapt their communication as a capable human assistant would. How do we measure these capabilities in a model or system? Proposals with a special focus on evaluating different facets of emotional and social intelligence, such as emotion recognition and comprehension, empathy, intent understanding, identity and personality, persuasion, inspiration, theory of mind, conversation management, and multimodal emotion recognition, are encouraged.
Agentic Behavior: Llama 3 has been integrated into Meta AI, Meta’s intelligent assistant. What datasets, environments, or other assets will the academic community need to assess progress agents’ abilities to successfully complete tasks of interest for users? Proposals with a focus on taking actions in web, desktop, and app environments, continuous training, steerability (especially for conversational tone and safety settings), personalization, and retrieval capabilities in the system, are encouraged.

Requirements

Prior to submitting a proposal, please confirm that your institution will agree to a contract for sponsored research from Meta. The agreement will operate under “Open Science” terms meaning that all research results from this grant will be made available for publication or public release into the public domain. Application materials include:

A summary of the project (2-4 pages), in English, explaining the area of focus, a description of techniques, any relevant prior work, and a timeline with milestones and expected outcomes;
A draft budget description (1 page) including an approximate cost of the award and explanation of how funds would be spent (including a line for overhead and the required travel to the Open Innovation AI Research Community Workshop in London, from October 29th - October 30th, 2024*)
Curriculum Vitae for all project participants; and
Organization details; this will include tax information and administrative contact details.

Eligibility

Proposals must comply with applicable U.S. and international laws, regulations, and policies;
Applicants must be current full-time faculty at an accredited academic institution that awards research degrees to PhD students;
Applicants must be the Principal Investigator on any resulting award; please note that only one grant recipient will be awarded per proposal.
Meta cannot consider proposals submitted, prepared or to be carried out by individuals residing in, or affiliated with an academic institution located in, a country or territory subject to comprehensive U.S. trade sanctions; and
Government officials (excluding faculty and staff of public universities, to the extent they may be considered government officials), political figures, and politically affiliated businesses (all as determined by Meta in its sole discretion) are not eligible.
The research period for all proposals is one year, or 12 months from the date of receiving the grant.
Grant recipients must be able to attend the Open Innovation AI Research Community Research Workshop in London, UK from October 29th - 30th, 2024.

Note: Proposals can utilize any permissively licensed LLM and are not limited to a requirement to use Meta’s Llama models in the work.

Evaluation

All eligible Proposals will be reviewed and rated by Meta during September 2024.

Technically challenging: Benchmarks submitted need to be challenging and applicable to not only current models, but new models to come in the future.

Impact: Proposal articulates a vision and plan for how the proposed solution measures qualities needed for complex reasoning, emotional and social intelligence, or agentic behavior.

Reach: Where applicable, Proposal articulates how the proposed solution can scale to multiple languages or multiple modalities.

FAQ

What is the Open Innovation AI Research Community?

The Open Innovation AI Research Community (“Research Community”) is a program for academic researchers, designed to foster collaboration and knowledge-sharing in the field of artificial intelligence. By joining this community, participants will have the chance to contribute to a research agenda that addresses the most pressing challenges in the field, and work together to develop innovative solutions that promote responsible and safe AI practices. We believe that by bringing together diverse perspectives and expertise, we can accelerate the pace and progress in AI research. For more information, please visit our webpage.

What is the Open Innovation AI Research Community Research Workshop?*

Every year, the Research Community comes together for an annual workshop - this year, the workshop will be held at Meta’s London offices from October 29th - 30th, 2024. Our goals are to share the research findings within the community, provide an opportunity for new collaborations, and network with other researchers. Grant recipients are required to attend to present their research and their proposals at the workshop. Please make sure to include travel for this workshop into your budget.

If you are not able to attend this workshop, please include a note in your proposal or identify an alternative member from your team who could represent your project at the workshop.

Can I partner with researchers at other universities for this proposal?

Yes, you may partner with researchers at other universities in your proposal. Please note however, that grants are only awarded to the organization employing the Principal Investigator of the proposal and cannot be split.

More information on Meta’s latest LLM launch

Launched in July 2024, Llama 3.1 is the latest version of Meta’s large language models (LLM). Llama 3.1 405B is an openly accessible model that excels at language nuances, contextual understanding, and complex tasks like translation and dialogue generation. With Llama 3.1, we’re encouraging developers to create custom agents, introducing new security and safety tools, and unlocking new workflows like synthetic data generation and model distillation. For more information on Llama 3.1, please visit our webpage.

Please confirm your identity

Please confirm your identity

Large Language Model (LLM) Evaluation Research Grant

Overview

Research Grant Finalists

Large Language Model (LLM) Research Grant Finalists