Large Language Model (LLM) Evaluation Research Grant

Areas of Interest

Meta encourages innovative proposals that will generate novel, challenging, ground truth benchmarks and evaluations, both for pre-training and post-training. We are specifically seeking benchmarks with human-generated data (we will disqualify synthetically generated data unless it has been disclosed and the proposal has a detailed explanation justifying the quality and usefulness of it).

Knowing that there are more than 7,000 known languages in the world - and LLMs are just beginning to be available and trusted in a fraction of them - we want to understand how we can scalably extend the coverage for capabilities we care about into many languages . How can we make sure models in different languages align with human values in a culturally sensitive way? How can we measure capabilities that we care about? Proposals that can scalably extend to cover multiple languages or modalities will be prioritized.

We encourage submissions that utilize evaluations in the following areas:

1. Complex Reasoning: As AI advances and becomes increasingly capable of interacting with humans and taking actions on their behalf, we need methods for understanding their reasoning and planning capabilities. How would we measure different types of intelligence and reasoning effectively as these models approach and/or surpass human performance? Proposals with a special focus on complex reasoning are encouraged, including a focus on complex mathematical reasoning, tool integration, hierarchical planning, memory and infinite context window, System 1 vs System 2 reasoning, multi-step deduction and induction, analogical thinking, and narrative analysis.
2. Emotional and Social Intelligence: If AI models and agent systems of the future are expected to interact with humans effectively, they will need not only cognitive intelligence but also emotional and social intelligence—the ability to understand human emotions and sentiments, respond emphatically, "read the room", and generally adapt their communication as a capable human assistant would. How do we measure these capabilities in a model or system? Proposals with a special focus on evaluating different facets of emotional and social intelligence, such as emotion recognition and comprehension, empathy, intent understanding, identity and personality, persuasion, inspiration, theory of mind, conversation management, and multimodal emotion recognition, are encouraged.
3. Agentic Behavior: Llama 3 has been integrated into Meta AI, Meta’s intelligent assistant. What datasets, environments, or other assets will the academic community need to assess progress agents’ abilities to successfully complete tasks of interest for users? Proposals with a focus on taking actions in web, desktop, and app environments, continuous training, steerability (especially for conversational tone and safety settings), personalization, and retrieval capabilities in the system, are encouraged.