RecSys Challenge 2024

Sponsored by

Sponsor Workshop - May 27th from 2:00 to 4:00 pm (UTC+2)

We are pleased to invite you to the RecSys 24 Challenge Workshop, which will be held virtually on Monday, May 27th from 2:00 to 4:00 pm (UTC+2). This workshop aims to introduce the organizers and share the motivation behind this year's competition. During the workshop, we will provide insights into the objectives and structure of the challenge, and we will address any questions or concerns you may have in an open Q&A session. This will be a great opportunity to interact with the organizers and fellow participants, gaining a deeper understanding of the competition and how to make the most of your participation.
To register for the workshop, please follow this link:

We look forward to your participation and hope to see you there!

About

This year's challenge focuses on online news recommendation, addressing both the technical and normative challenges inherent in designing effective and responsible recommender systems for news publishing. The challenge will delve into the unique aspects of news recommendation, including modeling user preferences based on implicit behavior, accounting for the influence of the news agenda on user interests, and managing the rapid decay of news items. Furthermore, our challenge embraces the normative complexities, involving investigating the effects of recommender systems on the news flow and whether they resonate with editorial values.

Organizers

The challenge is organized by Ekstra Bladet and JP/Politikens Hus A/S ("Ekstra Bladet"), Johannes Kruse1,2, Kasper Lindskow1, Anshuk Uppal2, Michael Riis Andersen2, Jes Frellsen2, Marco Polignano3, Claudio Pomo4 and Abhishek Srivastava5 based on the data provided by Ekstra Bladet.
  1. Ekstra Bladet / JP/Politikens Hus A/S
  2. Technical University of Denmark
  3. University of Bari Aldo Moro, Italy
  4. Politecnico di Bari, Italy
  5. IIM Visakhapatnam, India
If you have any questions, suggestions, or are experiencing any issues, do not hesitate to reach out:
  • Johannes.Kruse@jppol.dk

Challenge Task

The Ekstra Bladet RecSys Challenge aims to predict which article a user will click on from a list of articles that were seen during a specific impression. Utilizing the user's click history, session details (like time and device used), and personal metadata (including gender and age), along with a list of candidate news articles listed in an impression log, the challenge's objective is to rank the candidate articles based on the user's personal preferences. This involves developing models that encapsulate both the users and the articles through their content and the users' interests. The models are to estimate the likelihood of a user clicking on each article by evaluating the compatibility between the article's content and the user's preferences. The articles are ranked based on these likelihood scores, and the precision of these rankings is measured against the actual selections made by users.

Evaluation

To evaluate the models, we use several standard metrics in the recommendation field, including the area under the ROC curve (AUC), mean reciprocal rank (MRR), and normalized discounted cumulative gain (nDCG@K) for K shown recommendations. To address the normative complexities inherent in news recommendations, the test set incorporates samples specifically designed to assess models based on normative properties. This includes evaluating models on Beyond-Accuracy Objectives, such as intra-list diversity, serendipity, novelty, coverage, among others. The final result is the average of these metrics across all impression logs.

  • The primary metric for the challenge is AUC.

    • Prizes

    The top three teams will receive exciting cash prizes: $3,500 for first place, $2,500 for second, and $1,500 for third. Additionally, a special $2,500 prize will be awarded to the best academic team.


    Timeline

    The table below presents the comprehensive timeline and critical deadlines relevant to the challenge. It's essential to note that all listed dates and times are based on the Anywhere on Earth (AoE) timezone, marked at 23:59:59.

    When? What?
    Mar. 8, 2024 Start RecSys Challenge
    Release dataset
    Mar. 25, 2024 Submission System Open
    Apr. 4, 2024 Leaderboard live
    Jun. 21, 2024 End RecSys Challenge
    Jun. 24, 2024 Final Leaderboard & Winners
    EasyChair open for submissions
    Jul. 1, 2024 Code Upload
    Upload code of the final predictions
    Jul. 15, 2024 Paper Submission Due
    Aug. 3, 2024 Paper Acceptance Notifications
    Aug. 29, 2024 Camera-Ready Papers
    Oct. 14, 2024 RecSys Challenge Workshop
    @ACM RecSys 2024

    Dataset

    The Ekstra Bladet News Recommendation Dataset (EB-NeRD) is a large-scale Danish dataset created by Ekstra Bladet to support advancements and benchmarking in news recommendation research. EB-NeRD comprises over 2.3 million users and more than 380 million impression logs from Ekstra Bladet. Alongside, we offer a collection of more than 125,000 news articles news articles, enriched with textual content features such as titles, abstracts, and bodies. This enables text features in a low-resource language as context for recommender systems.

    For more details about the dataset and format:

    Dataset Description

    Download

    EB-NeRD is free to download for research purposes under General License Terms (“License Terms”). Before you download the dataset, please read the License Terms and click below button to confirm that you agree to them.

    I have read and accepted the License Terms

    In addition, to help researchers become familiar with our data and run quick experiments, we are releasing a demo and a small version of the EB-NeRD by randomly sampling 5,000 and 50,000 users and their behavior logs from the full dataset.

    Also, to get started, we have assembled a toolkit featuring a range of established news and general recommendation methods.

    EBRec


    Registration

    The Challenge's evaluation system will be hosted on the open-source platform Codabench. Please read the Challenge's Terms and Conditions. To register please follow:

    Register

    The Codabench website is occasionally offline for maintenance to ensure optimal performance. To stay updated on maintenance schedules, please visit the CodaLab Competitions Google Group. Thank you for your understanding and continued support!


    Submission Guidelines

    We invite researchers and practitioners to submit their work for the RecSys Challenge workshop. Note, more information, including the submission process and the program committee will be provided later, but here are the key details for submissions:

    • Format and Templates: We will follow the format and templates from previous years and all submissions will be pre-reviewed.
    • Winning Teams: The winning teams must submit papers and sign up for the workshop.
    • Feature Utilization: Solutions that utilize all features, including those that may yield information not available in a live setup, and report results both with and without these features (as discussed in the thread: link).
    The topics of interest include, but are not limited to:
    • Benchmarking and evaluation of recommender systems on EB-NeRD
    • Novel model architectures for news recommendation
    • Dataset analyses and preprocessing techniques
    • Contributions focused on beyond accuracy, such as fairness, diversity, coverage, etc.
    • Scalability and efficiency of recommendation algorithms
    • Cross-domain and multi-modal recommendations


    Leaderboard

    Updated: May 16, 2024.
    To be added to the Academic Teams' leaderboard, please fill out the Google Form: Academic Leaderboard.

    Search for multiple teams at once using commas (e.g., "Team1, Team2, ..., TeamN").
    Rank Team AUC MRR NDCG@5 NDCG@10