Publications | Maliheh Izadi, PhD

2026

Benchmarking AI Models in Software Engineering: A Review, Search Tool, and Unified Approach for Elevating Benchmark Quality

Roham Koohestani, Philippe Bekker, and Maliheh Izadi

IEEE Transactions on Software Engineering (TSE), 2026

Abs HTML

Benchmarks are essential for consistent evaluation and reproducibility. The integration of Artificial Intelligence into Software Engineering (AI4SE) has given rise to numerous benchmarks for tasks such as code generation and bug fixing. However, this surge presents challenges: (1) scattered benchmark knowledge across tasks, (2) difficulty in selecting relevant benchmarks, (3) the absence of a uniform standard for benchmark development, and (4) limitations of existing benchmarks. In this paper, we review 173 studies and identify 204 AI4SE benchmarks. We classify these benchmarks, analyze their limitations, and expose gaps in practices. Based on our review, we created BenchScout, a semantic search tool to find relevant benchmarks, using automated clustering of the contexts from associated studies. We conducted a user study with 22 participants to evaluate BenchScout’s usability, effectiveness, and intuitiveness which resulted in average scores of 4.5, 4.0, and 4.1 out of 5. To advance benchmarking standards, we propose BenchFrame, a unified method to enhance benchmark quality. As a case study, we applied BenchFrame to the HumanEval benchmark and addressed its main limitations. This led to HumanEvalNext, featuring (1) corrected errors, (2) improved language conversion, (3) expanded test coverage, and (4) increased difficulty. We then evaluated ten state-of-the-art code language models on HumanEval, HumanEvalPlus, and HumanEvalNext. On HumanEvalNext, models showed a pass@1 score reduction of 31.22% and 19.94% compared to HumanEval and HumanEvalPlus, respectively.
Bridging Developer Needs and Feasible Features for AI Assistants in IDEs

Agnia Sergeyuk, Ekaterina Koshchenko, Ilya Zakharov, Timofey Bryksin, and Maliheh Izadi

In IEEE/ACM 48th International Conference on Software Engineering (ICSE), Industry track, 2026

Abs

Despite the increasing presence of AI assistants in Integrated Development Environments, it remains unclear what developers actually need from these tools and which features are likely to be implemented in practice. To investigate this gap, we conducted a two-phase study. First, we interviewed 35 professional developers from three user groups (Adopters, Churners, and Non-Users) to uncover unmet needs and expectations. Our analysis revealed five key areas: Technology Improvement, Interaction, and Alignment, as well as Simplifying Skill Building, and Programming Tasks. We then examined the feasibility of addressing selected needs through an internal prediction market involving 102 practitioners. The results demonstrate a strong alignment between the developers’ needs and the practitioners’ judgment for features focused on implementation and context awareness. However, features related to proactivity and maintenance remain both underestimated and technically unaddressed. Our findings reveal gaps in current AI support and provide practical directions for developing more effective and sustainable in-IDE AI systems.
Code4MeV2: a Research-oriented Code-completion Platform

Roham Koohestani, Parham Bateni, Aydin Ebrahimi, Behdad Etezadi, Kiarash Karimi, and Maliheh Izadi

In IEEE/ACM 48th International Conference on Software Engineering (ICSE), Demo track, 2026

Abs HTML

The adoption of AI-powered code completion tools in software development has increased substantially, yet the user interaction data produced by these systems remain proprietary within large corporations. This creates a barrier for the academic community, as researchers must often develop dedicated platforms to conduct studies on human–AI interaction, making reproducible research and large-scale data analysis impractical. In this work, we introduce Code4MeV2, a research-oriented, open-source code completion plugin for JetBrains IDEs, as a solution to this limitation. Code4MeV2 is designed using a client–server architecture and features inline code completion and a context-aware chat assistant. Its core contribution is a modular and transparent data collection framework that gives researchers fine-grained control over telemetry and context gathering. Code4MeV2 achieves industry-comparable performance in terms of code completion, with an average latency of 200 ms. We assess our tool through a combination of an expert evaluation and a user study with eight participants. Feedback from both researchers and daily users highlights its informativeness and usefulness. We invite the community to adopt and contribute to this tool. More information about the tool can be found at https://app.code4me.me.
Human-AI Experience in Integrated Development Environments: A Systematic Literature Review

Agnia Sergeyuk, Ilya Zakharov, Ekaterina Koshchenko, and Maliheh Izadi

Empricial Software Engineering Journal (EMSE), 2026

Abs

The integration of AI into IDEs is reshaping software development, fundamentally altering how developers interact with their tools. This shift marks the emergence of Human-AI Experience in Integrated Development Environment (in-IDE HAX), a field that explores the evolving dynamics of Human-Computer Interaction in AI-assisted coding environments. Despite rapid adoption, research on in-IDE HAX remains fragmented which highlights the need for a unified overview of current practices, challenges, and opportunities. To provide a structured overview of existing research, we conduct a systematic literature review of 89 studies, summarizing current findings and outlining areas for further investigation. Our findings reveal that AI-assisted coding enhances developer productivity but also introduces challenges, such as verification overhead, automation bias, and over-reliance, particularly among novice developers. Furthermore, concerns about code correctness, security, and maintainability highlight the urgent need for explainability, verification mechanisms, and adaptive user control. Although recent advances have driven the field forward, significant research gaps remain, including a lack of longitudinal studies, personalization strategies, and AI governance frameworks. This review provides a foundation for advancing in-IDE HAX research and offers guidance for responsibly integrating AI into software development.
Developer Interaction Patterns with Proactive AI: A Five-Day Field Study

Nadine Kuo, Agnia Sergeyuk, Valerie Chen, and Maliheh Izadi

In ACM Conference on Intelligent User Interfaces (ACM IUI), 2026
Are Agents Just Automata? On the Formal Equivalence Between Agentic AI and the Chomsky Hierarchy

Roham Koohestani, Ziyou Li, Anton Podkopaev, and Maliheh Izadi

arXiv, 2026

Abs

This paper establishes a formal equivalence between the architectural classes of modern agentic AI systems and the abstract machines of the Chomsky hierarchy. We posit that the memory architecture of an AI agent is the definitive feature determining its computational power and that it directly maps it to a corresponding class of automaton. Specifically, we demonstrate that simple reflex agents are equivalent to Finite Automata, hierarchical task-decomposition agents are equivalent to Pushdown Automata, and agents employing readable/writable memory for reflection are equivalent to TMs. This Automata-Agent Framework provides a principled methodology for right-sizing agent architectures to optimize computational efficiency and cost. More critically, it creates a direct pathway to formal verification, enables the application of mature techniques from automata theory to guarantee agent safety and predictability. By classifying agents, we can formally delineate the boundary between verifiable systems and those whose behavior is fundamentally undecidable. We address the inherent probabilistic nature of LLM-based agents by extending the framework to probabilistic automata that allow quantitative risk analysis. The paper concludes by outlining an agenda for developing static analysis tools and grammars for agentic frameworks.
Does In-IDE Calibration of Large Language Models work at Scale?

Roham Koohestani, Agnia Sergeyuk, David Gros, Claudio Spiess, Sergey Titov, Prem Devanbu , and 1 more author

arXiv, 2026

Abs

The introduction of large language models into integrated development environments (IDEs) is revolutionizing software engineering, yet it poses challenges to the usefulness and reliability of Artificial Intelligence-generated code. Post-hoc calibration of internal model confidences aims to align probabilities with an acceptability measure. Prior work suggests calibration can improve alignment, but at-scale evidence is limited. In this work, we investigate the feasibility of applying calibration of code models to an in-IDE context. We study two aspects of the problem: (1) the technical method for implementing confidence calibration and improving the reliability of code generation models, and (2) the human-centered design principles for effectively communicating reliability signal to developers. First, we develop a scalable and flexible calibration framework which can be used to obtain calibration weights for open-source models using any dataset, and evaluate whether calibrators improve the alignment between model confidence and developer acceptance behavior. Through a large-scale analysis of over 24 million real-world developer interactions across multiple programming languages, we find that a general, post-hoc calibration model based on Platt-scaling does not, on average, improve the reliability of model confidence signals. We also find that while dynamically personalizing calibration to individual users can be effective, its effectiveness is highly dependent on the volume of user interaction data. Second, we conduct a multi-phase design study with 3 expert designers and 153 professional developers, combining scenario-based design, semi-structured interviews, and survey validation, revealing a clear preference for presenting reliability signals via non-numerical, color-coded indicators within the in-editor code generation workflow.

2025

Evaluating Large Language Models for Functional and Maintainable Code in Industrial Settings: A Case Study at ASM

Yash Mundhra, Max Valk, and Maliheh Izadi

In IEEE/ACM International Conference on Automated Software Engineering (ASE), Industry Track, 2025

Abs

Large language models have shown impressive performance in various domains, including code generation across diverse open-source domains. However, their applicability in proprietary industrial settings, where domain-specific constraints and code interdependencies are prevalent, remains largely unexplored. We present a case study conducted in collaboration with the leveling department at ASML to investigate the performance of LLMs in generating functional, maintainable code within a closed, highly specialized software environment. We developed an evaluation framework tailored to ASML’s proprietary codebase and introduced a new benchmark. Additionally, we proposed a new evaluation metric, build@k, to assess whether LLM-generated code successfully compiles and integrates within real industrial repositories. We investigate various prompting techniques, compare the performance of generic and code-specific LLMs, and examine the impact of model size on code generation capabilities, using both match-based and execution-based metrics. The findings reveal that prompting techniques and model size have a significant impact on output quality, with few-shot and chain-of-thought prompting yielding the highest build success rates. The difference in performance between the code-specific LLMs and generic LLMs was less pronounced and varied substantially across different model families.
Prompt-with-Me: in-IDE Structured Prompt Management for LLM-Driven Software Engineering

Ziyou Li, Agnia Sergeyuk, and Maliheh Izadi

In IEEE/ACM International Conference on Automated Software Engineering (ASE), Industry Track, 2025

Abs HTML

Large Language Models are transforming software engineering, yet prompt management in practice remains ad hoc, hindering reliability, reuse, and integration into industrial workflows. We present Prompt-with-Me, a practical solution for structured prompt management embedded directly in the development environment. The system automatically classifies prompts using a four-dimensional taxonomy encompassing intent, author role, software development lifecycle stage, and prompt type. To enhance prompt reuse and quality, Prompt-with-Me suggests language refinements, masks sensitive information, and extracts reusable templates from a developer’s prompt library. Our taxonomy study of 1,108 real-world prompts demonstrates that modern LLMs can accurately classify software engineering prompts. Furthermore, our user study with 11 participants shows strong developer acceptance, with high usability (Mean SUS=73), low cognitive load (Mean NASA-TLX=21), and reported gains in prompt quality and efficiency through reduced repetitive effort. Lastly, we offer actionable insights for building the next generation of prompt management and maintenance tools for software engineering workflows.
TreeRanker: Fast and Model-agnostic Ranking System for Code Suggestions in IDEs

Daniele Cipollone, Egor Bogomolov, Arie Deursen, and Maliheh Izadi

In IEEE/ACM International Conference on Automated Software Engineering (ASE), Industry Track, 2025

Abs HTML

Token-level code completion is one of the most critical features in modern Integrated Development Environments (IDEs). It assists developers by suggesting relevant identifiers and APIs during coding. While completions are typically derived from static analysis, their usefulness depends heavily on how they are ranked, as correct predictions buried deep in the list are rarely seen by users. Most current systems rely on hand-crafted heuristics or lightweight machine learning models trained on user logs, which can be further improved to capture context information and generalize across projects and coding styles. In this work, we propose a new scoring approach to ranking static completions using language models in a lightweight and model-agnostic way. Our method organizes all valid completions into a prefix tree and performs a single greedy decoding pass to collect token-level scores across the tree. This enables a precise token-aware ranking without needing beam search, prompt engineering, or model adaptations. The approach is fast, architecture-agnostic, and compatible with already deployed models for code completion. These findings highlight a practical and effective pathway for integrating language models into already existing tools within IDEs, and ultimately providing smarter and more responsive developer assistance.
Code Red! On the Harmfulness of Applying Off-the-shelf Large Language Models to Programming Tasks

Ali Al-kaswan, Sebastian Deatc, Begum Koc, Arie Deursen, and Maliheh Izadi

In ACM International Conference on the Foundations of Software Engineering (FSE), Main Track, 2025

Abs

Nowadays, developers increasingly rely on solutions powered by Large Language Models (LLM) to assist them with their coding tasks. This makes it crucial to align these tools with human values to prevent malicious misuse. In this paper, we propose a comprehensive framework for assessing the potential harmfulness of LLMs within the software engineering domain. We begin by developing a taxonomy of potentially harmful software engineering scenarios and subsequently, create a dataset of prompts based on this taxonomy. To systematically assess the responses, we design and validate an automatic evaluator that classifies the outputs of a variety of LLMs both open-source and closed-source models, as well as general-purpose and code-specific LLMs. Furthermore, we investigate the impact of models’ size, architecture family, and alignment strategies on their tendency to generate harmful content. The results show significant disparities in the alignment of various LLMs for harmlessness. We find that some models and model families, such as Openhermes, are more harmful than others and that code-specific models do not perform better than their general-purpose counterparts. Notably, some fine-tuned models perform significantly worse than their base-models due to their design choices. On the other side, we find that larger models tend to be more helpful and are less likely to respond with harmful information. These results highlight the importance of targeted alignment strategies tailored to the unique challenges of software engineering tasks and provide a foundation for future work in this critical area.
Leveraging large language models for enhancing the understandability of generated unit tests

Amirhossein Deljouyi, Roham Koohestani, Maliheh Izadi, and Andy Zaidman

In IEEE/ACM 47th International Conference on Software Engineering (ICSE), Main Track, 2025

Abs

Automated unit test generators, particularly search-based software testing tools like EvoSuite, are capable of generating tests with high coverage. Although these generators alleviate the burden of writing unit tests, they often pose challenges for software engineers in terms of understanding the generated tests. To address this, we introduce UTGen, which combines search-based software testing and large language models to enhance the understandability of automatically generated test cases. We achieve this enhancement through contextualizing test data, improving identifier naming, and adding descriptive comments. Through a controlled experiment with 32 participants from both academia and industry, we investigate how the understandability of unit tests affects a software engineer’s ability to perform bug-fixing tasks. We selected bug-fixing to simulate a real-world scenario that emphasizes the importance of understandable test cases. We observe that participants working on assignments with UTGen test cases fix up to 33% more bugs and use up to 20% less time when compared to baseline test cases. From the post-test questionnaire, we gathered that participants found that enhanced test names, test data, and variable names improved their bug-fixing process.
HyperSeq: A Hyper-Adaptive Representation for Predictive Sequencing of States

Roham Koohestani, and Maliheh Izadi

In ACM International Conference on the Foundations of Software Engineering (FSE), New Idea and Vision Track, 2025

Abs

In the rapidly evolving world of software development, the surge in developers’ reliance on AI-driven tools has transformed Integrated Development Environments into powerhouses of advanced features. This transformation, while boosting developers’ productivity to unprecedented levels, comes with a catch: increased hardware demands for software development. Moreover, the significant economic and environmental toll of using these sophisticated models necessitates mechanisms that reduce unnecessary computational burdens. We propose HyperSeq - Hyper-Adaptive Representation for Predictive Sequencing of States - a novel, resource-efficient approach designed to model developers’ cognitive states. HyperSeq facilitates precise action sequencing and enables real-time learning of user behavior. Our preliminary results show how HyperSeq excels in forecasting action sequences and achieves remarkable prediction accuracies that go beyond 70%. Notably, the model’s online-learning capability allows it to substantially enhance its predictive accuracy in a majority of cases and increases its capability in forecasting next user actions with sufficient iterations for adaptation. Ultimately, our objective is to harness these predictions to refine and elevate the user experience dynamically within the IDE.
A Multi-agent Onboarding Assistant based on Large Language Models, Retrieval Augmented Generation, and Chain-of-Thought

Andrei Cristian Ionescu, Sergey Titov, and Maliheh Izadi

In ACM International Conference on the Foundations of Software Engineering (FSE), Demo track, 2025
Enhancing Human-IDE Interaction in the SDLC using LLM-based Mediator Agents

Ziyou Li, and Maliheh Izadi

In The 1st International Workshop on AI-Augmented SDLC, co-located with (FSE), 2025

Abs

Large Language Model (LLM)-based autonomous agents provide state-of-the-art performance in various tasks throughout the Software Development Lifecycle (SDLC). Though the applications employing the agents can be implemented in Integrated Development Environments (IDEs), the standalone applications are not optimal for human programmers’ utilization because of the cognitive burden caused by learning and utilizing different tools required throughout the SDLC. This paper proposes a framework of a mediator agent acting as the sole medium between programmer, IDE tools, agentic tools, and external multi-agent systems. The framework aims to simplify tool usage and automate the management of in-IDE tools and other agentic tools that can be implemented in the future. By implementing a unified mediator agent, this project aims to set the stage for human-computer interaction in the era of LLM-based autonomous agents, improve programmer productivity, and advance software engineering automation.
A Qualitative Investigation into LLM-Generated Multilingual Code Comments and Automatic Evaluation Metrics

Jonathan Katzy, Yongcheng Huang, Gopal-Raj Panchu, Maksym Ziemlewski, Paris Loizides, Sander Vermeulen , and 2 more authors

In The International Conference on Predictive Models and Data Analytics in Software Engineering (PROMISE), 2025

Abs

Large Language Models are essential coding assistants, yet their training is predominantly English-centric. In this study, we evaluate the performance of code language models in non-English contexts, identifying challenges in their adoption and integration into multi-lingual workflows. We conduct an open-coding study to analyze errors in code comments generated by five state-of-the-art code models, CodeGemma, CodeLlama, CodeQwen1.5, GraniteCode, and StarCoder2 across five natural languages: Chinese, Dutch, English, Greek, and Polish. Our study yields a dataset of 12,500 labeled generations, which we publicly release. We then assess the reliability of standard metrics in capturing comment correctness across languages and evaluate their trustworthiness as judgment criteria. Through our open-coding investigation, we identified a taxonomy of 26 distinct error categories in model-generated code comments. They highlight variations in language cohesion, informativeness, and syntax adherence across different natural languages. Our analysis shows that while these models frequently produce partially correct comments, modern neural metrics fail to reliably differentiate meaningful completions from random noise. Notably, the significant score overlap between expert-rated correct and incorrect comments calls into question the effectiveness of these metrics in assessing generated comments.
Automated Attention Pattern Discovery at Scale in Large Language Models

Jonathan Katzy, Razvan Mihai Popescu, Arie Deursen, and Maliheh Izadi

In Mechanistic Interpretability Workshop (co-located with NeurIPS), 2025

Abs
Long Code Arena: a Set of Benchmarks for Long-context Code Models

Egor Bogomolov, Aleksandra Eliseeva, Timur Galimzyanov, Evgeniy Glukhov, Anton Shapkin, Maria Tigina , and 5 more authors

ArXiv, 2025

Abs

Nowadays, the fields of code and natural language processing are evolving rapidly. In particular, models become better at processing long context windows - supported context sizes have increased by orders of magnitude over the last few years. However, there is a shortage of benchmarks for code processing that go beyond a single file of context, while the most popular ones are limited to a single method. With this work, we aim to close this gap by introducing Long Code Arena, a suite of six benchmarks for code processing tasks that require project-wide context. These tasks cover different aspects of code processing: library-based code generation, CI builds repair, project-level code completion, commit message generation, bug localization, and module summarization. For each task, we provide a manually verified dataset for testing, an evaluation suite, and open-source baseline solutions based on popular LLMs to showcase the usage of the dataset and to simplify adoption by other researchers. We publish the benchmark page on HuggingFace Spaces with the leaderboard, links to HuggingFace Hub for all the datasets, and link to the GitHub repository with baselines: https://huggingface.co/spaces/JetBrains-Research/long-code-arena.
The Heap: A Contamination-Free Multilingual Code Dataset for Evaluating Large Language Models

Jonathan Katzy, Razvan Mihai Popescu, Arie Deursen, and Maliheh Izadi

In The 2nd ACM International Conference on AI Foundation Models and Software Engineering (FORGE), 2025

Abs

The recent rise in the popularity of large language models has spurred the development of extensive code datasets needed to train them. This has left limited code available for collection and use in the downstream investigation of specific behaviors, or evaluation of large language models without suffering from data contamination. To address this problem, we release The Heap, a large multilingual dataset covering 57 programming languages that has been deduplicated with respect to other open datasets of code, enabling researchers to conduct fair evaluations of large language models without significant data cleaning overhead.
How Much Do Code Language Models Remember? An Investigation on Data Extraction Attacks before and after Fine-tuning

Fabio Salerno, Ali Al-Kaswan, and Maliheh Izadi

In IEEE/ACM 22th International Conference on Mining Software Repositories (MSR), 2025

ACM Distinguished Paper Award 🏆 Abs

We received the ACM Distinguished Paper Award for this work.

Code language models, while widely popular, are often trained on unsanitized source code gathered from across the Internet. Previous work revealed that pre-trained models can remember the content of their training data and regurgitate them through data extraction attacks. Due to the large size of current models, only a few entities have the resources for pre-training such models. However, fine-tuning requires fewer resources and is increasingly used by both small and large entities for its effectiveness on specialized data. Such small curated data for fine-tuning might contain sensitive information or proprietary assets. In this study, we attack both pre-trained and fine-tuned code language models to investigate the extent of data extractability. We first develop a custom benchmark to assess the vulnerability of both pre-training and fine-tuning samples to extraction attacks. Our findings reveal that 54.9% of extractable pre-training data could be retrieved from StarCoder2-15B, whereas this number decreased to 23.5% after fine-tuning. This indicates that fine-tuning reduces the extractability of pre-training data. However, compared to larger models, fine-tuning smaller models increases their vulnerability to data extraction attacks on fine-tuning data. Given the potential sensitivity of fine-tuning data, this can lead to more severe consequences. Lastly, we also manually analyzed 2000 extractable samples before and after fine-tuning. We also found that data carriers and licensing information are the most likely data categories to be memorized from pre-trained and fine-tuned models, while the latter is the most likely to be forgotten after fine-tuning.
Rethinking IDE Customization for Enhanced HAX: A Hyperdimensional Perspective

Roham Koohestani, and Maliheh Izadi

In The 2nd Workshop on Integrated Development Environments (IDE), 2025

Abs

As Integrated Development Environments (IDEs) increasingly integrate Artificial Intelligence, Software Engineering faces both benefits like productivity gains and challenges like mismatched user preferences. We propose Hyper-Dimensional (HD) vector spaces to model Human-Computer Interaction, focusing on user actions, stylistic preferences, and project context. These contributions aim to inspire further research on applying HD computing in IDE design.
Automating the Detection of Code Vulnerabilities by Analyzing GitHub Issues

Daniele Cipollone, Changjie Wang, Mariano Scazzariello, Simone Ferlin, Maliheh Izadi, Dejan Kostic , and 1 more author

In The 2nd International Workshop on LLM4Code, co-located with ICSE, 2025

Abs

In today’s digital landscape, the importance of timely and accurate vulnerability detection has significantly increased. This paper presents a novel approach that leverages transformer-based models and machine learning techniques to automate the identification of software vulnerabilities by analyzing GitHub issues. We introduce a new dataset specifically designed for classifying GitHub issues relevant to vulnerability detection. We then examine various classification techniques to determine their effectiveness. The results demonstrate the potential of this approach for real-world application in early vulnerability detection, which could substantially reduce the window of exploitation for software vulnerabilities. This research makes a key contribution to the field by providing a scalable and computationally efficient framework for automated detection, enabling the prevention of compromised software usage before official notifications. This work has the potential to enhance the security of open-source software ecosystems.
Enhancing Large Language Model Integration in Integrated Development Environments

Daniele Cipollone

In ACM International Conference on the Foundations of Software Engineering (FSE), Doctoral Symposium, 2025

Abs

Large language models have emerged as one of the most impactful applications in software engineering. However, their integration into the software development ecosystem remains fragmented and suboptimal. This study aims to bridge this gap by adapting and fine-tuning existing large language models for real-world software development scenarios. Rather than building new models from scratch, our approach leverages advanced techniques that allow Large Language Models to interact seamlessly with existing development tools and Integrated Development Environments (IDEs). This research, conducted in collaboration with JetBrains, focuses on personalizing code generation by adjusting the model to the unique features of a project and the individual coding style of each developer.
Mediating between Human Programmers and Integrated Development Environments using LLM-based Agents

Ziyou Li

In ACM International Conference on the Foundations of Software Engineering (FSE), Doctoral Symposium, 2025

Abs

Large Language Model (LLM)-based autonomous agents provide state-of-the-art performance in various tasks throughout the software development lifecycle. Though the applications employing the agents can be implemented in Integrated Development Environments (IDEs), the standalone applications is not optimal for human programmers’ utilization. This doctoral project proposes a framework of an mediator agent acting as the sole medium between programmer, IDE tools, agentic tools, and external multi-agent systems. The framework aims to simplify tool usage and automate management of in-IDE tools and other agentic tools that can be implemented in the future. By implementing a unified mediator agent, this project expects to set the stage of human-computer interaction in the era of LLM-based autonomous agents, enhance programmer productivity, and advance in software engineering automation.
In-IDE Proactive AI Assistant for Code Optimisation

Nadine Kuo

2025

Abs
Enhancing Issue Tracking Efficiency with AI-Driven Natural Language Processing: Improving Classification, Association and Resolution at NXP

Venelina Pocheva

2025
LLMs for Domain Specific Code Generation at ASML

Yash Mundhra

2025
Dataset Development for LLMs4Code Licensing, Contamination, and Reproducibility Challenges

Razvan Popescu

2025

Abs

2024

An Exploratory Investigation into Code License Infringements in Large Language Model Training Datasets

Jonathan Katzy, Razvan Popescu, Arie Van Deursen, and Maliheh Izadi

In The 2nd ACM International Conference on AI Foundation Models and Software Engineering (FORGE), 2024

Abs

Does the training of large language models potentially infringe upon code licenses? Furthermore, are there any datasets available that can be safely used for training these models without violating such licenses? In our study, we assess the current trends in the field and the importance of incorporating code into the training of large language models. Additionally, we examine publicly available datasets to see whether these models can be trained on them without the risk of legal issues in the future. To accomplish this, we compiled a list of 53 large language models trained on file-level code. We then extracted their datasets and analyzed how much they overlap with a dataset we created, consisting exclusively of strong copyleft code.Our analysis revealed that every dataset we examined contained license inconsistencies, despite being selected based on their associated repository licenses. We analyzed a total of …
Generative AI in Software Engineering Must be Human-centered: The Copenhagen Manifesto

Daniel Russo, Sebastian Baltes, Niels Berkel, Paris Avgeriou, Fabio Calefato, Beatriz Cabrero-Daniel , and 29 more authors

Journal of Systems and Software, 2024
Creativity, Generative AI, and Software Development: A Research Agenda

Victoria Jackson, Bogdan Vasilescu, Daniel Russo, Paul Ralph, Maliheh Izadi, Rafael Prikladnicki , and 4 more authors

2024

Abs

Creativity has always been considered a major differentiator to separate the good from the great, and we believe the importance of creativity for software development will only increase as GenAI becomes embedded in developer tool-chains and working practices. This paper uses the McLuhan tetrad alongside scenarios of how GenAI may disrupt software development more broadly, to identify potential impacts GenAI may have on creativity within software development. The impacts are discussed along with a future research agenda comprising six connected themes that consider how individual capabilities, team capabilities, the product, unintended consequences, society, and human aspects can be affected.
A Transformer-based Approach for Smart Invocation of Automatic Code Completion

Aral Moor, Arie Deursen, and Maliheh Izadi

In The 1st ACM International Conference on AI-powered Software (AIware), 2024

ACM Distinguished Paper Award 🏆 Abs

We received the ACM Distinguished Paper Award for this work.

Transformer-based language models are highly effective for code completion, with much research dedicated to enhancing the content of these completions. Despite their effectiveness, these models come with high operational costs and can be intrusive, especially when they suggest too often and interrupt developers who are concentrating on their work. Current research largely overlooks how these models interact with developers in practice and neglects to address when a developer should receive completion suggestions. To tackle this issue, we developed a machine learning model that can accurately predict when to invoke a code completion tool given the code context and available telemetry data. To do so, we collect a dataset of 200k developer interactions with our cross-IDE code completion plugin and train several invocation filtering models. Our results indicate that our small-scale transformer model …
In-IDE Human-AI Experience in the Era of Large Language Models; a Literature Review

Agnia Sergeyuk, Sergey Titov, and Maliheh Izadi

In The 1st Workshop on Integrated Development Environments (IDE), 2024

Abs

Integrated Development Environments (IDEs) have become central to modern software development, especially with the integration of Artificial Intelligence (AI) to enhance programming efficiency and decision-making. The study of in-IDE Human-AI Experience is critical in understanding how these AI tools are transforming the software development process, impacting programmer productivity, and influencing code quality.We conducted a literature review to study the current state of in-IDE Human-AI Experience research, bridging a gap in understanding the nuanced interactions between programmers and AI assistants within IDEs. By analyzing 36 selected papers, our study illustrates three primary research branches: Design, Impact, and Quality of Interaction.The trends, challenges, and opportunities identified in this paper emphasize the evolving landscape of software development and inform future …
Investigating the performance of language models for completing code in functional programming languages: a haskell case study

Tim Van Dam, Frank Van Heijden, Philippe De Bekker, Berend Nieuwschepen, Marc Otten, and Maliheh Izadi

In The 1st ACM International Conference on AI Foundation Models and Software Engineering (FORGE), 2024

Abs

Language model-based code completion models have quickly grown in use, helping thousands of developers write code in many different programming languages. However, research on code completion models typically focuses on imperative languages such as Python and JavaScript, which results in a lack of representation for functional programming languages. Consequently, these models often perform poorly on functional languages such as Haskell. To investigate whether this can be alleviated, we evaluate the performance of two language models for code, CodeGPT and UniXcoder, on the functional programming language Haskell. We fine-tune and evaluate the models on Haskell functions sourced from a publicly accessible Haskell dataset on HuggingFace. Additionally, we manually evaluate the models using our novel translated HumanEval dataset. Our automatic evaluation shows that knowledge of …
Language models for code completion: A practical evaluation

Maliheh Izadi, Jonathan Katzy, Tim Van Dam, Marc Otten, Razvan Mihai Popescu, and Arie Van Deursen

In IEEE/ACM 46th International Conference on Software Engineering (ICSE), Main Track, 2024

Abs

Transformer-based language models for automatic code completion have shown great promise so far, yet the evaluation of these models rarely uses real data. This study provides both quantitative and qualitative assessments of three public code language models when completing real-world code. We first developed an open-source IDE extension, Code4Me, for the online evaluation of the models. We collected real auto-completion usage data for over a year from more than 1200 users, resulting in over 600K valid completions. These models were then evaluated using six standard metrics across twelve programming languages. Next, we conducted a qualitative study of 1690 real-world completion requests to identify the reasons behind the poor model performance. A comparative analysis of the models’ performance in online and offline settings was also performed, using benchmark synthetic datasets and two …
Traces of Memorisation in Large Language Models for Code

Ali Al-Kaswan, Maliheh Izadi, and Arie Van Deursen

In IEEE/ACM 46th International Conference on Software Engineering (ICSE), Main Track, 2024

Abs

Large language models have gained significant popularity because of their ability to generate human-like text and potential applications in various fields, such as Software Engineering. Large language models for code are commonly trained on large unsanitised corpora of source code scraped from the internet. The content of these datasets is memorised and can be extracted by ers with data extraction attacks. In this work, we explore memorisation in large language models for code and compare the rate of memorisation with large language models trained on natural language. We adopt an existing benchmark for natural language and construct a benchmark for code by identifying samples that are vulnerable to attack. We run both benchmarks against a variety of models, and perform a data extraction attack. We find that large language models for code are vulnerable to data extraction attacks, like their …
Maven Unzipped: Exploring the Impact of Library Packaging on the Ecosystem

Mehdi Keshani, Gideon Bot, Priyam Rungta, Maliheh Izadi, Arie Van Deursen, and Sebastian Proksch

In IEEE International Conference on Software Maintenance and Evolution (ICSME), 2024

Abs

MAVEN is a popular dependency management tool and ecosystem used by millions of developers. However, the over-whelming amount of available open-source software and the lack of proper ecosystem governance pose risks to the security and effectiveness of the ecosystem. This necessitates a comprehensive understanding of the ecosystem to guide future decision-making and promote effective practices. Despite numerous studies on aspects of Maven,such as vulnerabilities, breaking changes, and bloated dependencies, a knowledge gap concerning its overall state and health still exists. This gap impedes the adoption of effective practices, potentially impacting the productivity and efficiency of projects and the ecosystem as a whole. This paper explores the fundamental aspects of the Mavenecosystem. We investigate the packaging practices of Mavenlibraries with a focus on the content of the libraries, their …
Towards Safe, Secure, and Usable LLMs4Code

Ali Al-kaswan

In IEEE/ACM 46th International Conference on Software Engineering (ICSE), Doctoral Symposium, 2024

Abs

Large Language Models (LLMs) are gaining popularity in the field of Natural Language Processing (NLP) due to their remarkable accuracy in various NLP tasks. LLMs designed for coding are trained on massive datasets, which enables them to learn the structure and syntax of programming languages. These datasets are scraped from the web and LLMs memorise information in these datasets. LLMs for code are also growing, making them more challenging to execute and making users increasingly reliant on external infrastructure. We aim to explore the challenges faced by LLMs for code and propose techniques to measure and prevent memorisation. Additionally, we suggest methods to compress models and run them locally on consumer hardware.
Programming Language Models in Multilingual Settings

Jonathan Katzy

In IEEE/ACM 46th International Conference on Software Engineering (ICSE), Doctoral Symposium, 2024

Abs

Large language models have become increasingly utilized in programming contexts. However, due to the recent emergence of this trend, some aspects have been overlooked. We propose a research approach that investigates the inner mechanics of transformer networks, on a neuron, layer, and output representation level, to understand whether there is a theoretical limitation that prevents large language models from performing optimally in a multilingual setting. We propose to approach the investigation into the theoretical limitations, by addressing open problems in machine learning for the software engineering community. This will contribute to a greater understanding of large language models for programming-related tasks, making the findings more approachable to practitioners, and simply their implementation in future models.
Message from NLBSE 2024 Program Chairs

Maliheh Izadi, Andrea Di Sorbo, and Sebastiano Panichella

2024

Abs

Natural Language Processing (NLP) refers to the automated elaboration of human language, including both algorithms that take human-produced text as input and algorithms that produce natural-looking text as outputs. NLP is widely used to optimize many aspects of the software development process. Since natural language artifacts are used and reused during the software development life-cycle, the availability of natural language-based approaches and tools has led to improvements in the software process and product efficiency. Indeed, NLP approaches (including LLMs) have proven useful for retrieving key information from a wide range of structured or unstructured sources. Besides, they show promise for the automated generation of fine-grained source code documentation to ease program comprehension and maintenance activities. Literature has shown that many software engineering (SE)-related tasks …
Message from the Industry Track Co-Chairs

Maliheh Izadi, and Laura Moreno

2024

Abs

The MSR 2024 Industry Track is the venue to present and learn about the opportunities, challenges, and cutting-edge technology related to using data from software repositories in practice. For a long time, academic researchers in software engineering have been looking to learn and collaborate with practitioners. Our goal for the Industry Track is to be the space for a productive dialogue between software engineering researchers and practitioners, particularly those building tools for other software professionals.In this third edition of the Industry Track, we were looking to minimize the known barriers to industry participation in software engineering research conferences. We invited practitioners to submit an abstract, maximum one page long (plus up to one page of references), outlining a talk or a poster presentation.
The Potential of an Adaptive Computerized Dynamic Assessment Tutor in Diagnosing and Assessing Learners’ Listening Comprehension

Mehri Izadi, Maliheh Izadi, and Farrokhlagha Heidari

Education and Information Technologies, 2024

Abs

In today’s environment of growing class sizes due to the prevalence of online and e-learning systems, providing one-to-one instruction and feedback has become a challenging task for teachers. Anyhow, the dialectical integration of instruction and assessment into a seamless and dynamic activity can provide a continuous flow of assessment information for teachers to boost and individualize learning. In this regard, adaptive learning technology is one way to facilitate teacher-supported learning and personalize curriculum and learning experiences. This study aimed to investigate the potential of an adaptive Computerized Dynamic Assessment (C-DA) tool applicable as a language diagnostician and assistant. The study tried to get insight into 75 Iranian EFL learners’ listening development by focusing on the learning potential exhibited through learners’ assessment and the degree of internalization of mediation. To …
The Impact of Generative AI on Creativity in Software Development: A Research Agenda

Victoria Jackson, Bogdan Vasilescu, Daniel Russo, Paul Ralph, Maliheh Izadi, Rafael Prikladnicki , and 4 more authors

2024

Abs

As GenAI becomes embedded in developer toolchains and practices, and routine code is increasingly generated, human creativity will be increasingly important for generating competitive advantage. This paper uses the McLuhan tetrad alongside scenarios of how GenAI may disrupt software development more broadly, to identify potential impacts GenAI may have on creativity within software development. The impacts are discussed along with a future research agenda comprising five connected themes that consider how individual capabilities, team capabilities, the product, unintended consequences, and society. can be affected.
Meet Your Onboarding Buddy: A Smart, Adaptive, & Conversational LLM Assistant

Andrei Ionescu

2024

Abs
Beyond Acceptance Rates: The Impact of JetBrains AI Assistant and FLCC

Remco Schrijver

2024
Black-box Context-Aware Code Completion

Tim Dam

2024
AI for Software Engineering: Reviewing and Improving Benchmarking Practices

Philippe Bekker

2024
Building and Evaluating an LLM-based Code Completion Plugin

Frank Heijden

2024
Extracting Training Data from Fine-tuned Large Language Models

Fabio Salerno

2024

2023

On the Impact of Language Selection for Training and Evaluating Programming Language Models

Jonathan Katzy, Maliheh Izadi, and Arie Van Deursen

In IEEE 23rd International Working Conference on Source Code Analysis and Manipulation (SCAM), 2023

Abs

The recent advancements in Transformer-based Language Models have demonstrated significant potential in enhancing the multilingual capabilities of these models. The remarkable progress made in this domain not only applies to natural language tasks but also extends to the domain of programming languages. Despite the ability of these models to learn from multiple languages, evaluations typically focus on particular combinations of the same languages. In this study, we evaluate the similarity of programming languages by analyzing their representations using a CodeBERT-based model. Our experiments reveal that token representation in languages such as C++, Python, and Java exhibit proximity to one another, whereas the same tokens in languages such as Mathematica and R display significant dissimilarity. Our findings suggest that this phenomenon can potentially result in performance challenges …
Enriching Source Code with Contextual Data for Code Completion Models: An Empirical Study

Tim Dam, Maliheh Izadi, and Arie Deursen

In IEEE/ACM 20th International Conference on Mining Software Repositories (MSR), 2023

Abs

Transformer-based pre-trained models have recently achieved great results in solving many software engineering tasks including automatic code completion which is a staple in a developer’s toolkit. While many have striven to improve the code-understanding abilities of such models, the opposite – making the code easier to understand – has not been properly investigated. In this study, we aim to answer whether making code easier to understand through using contextual data improves the performance of pre-trained code language models for the task of code completion. We consider type annotations and comments as two common forms of additional contextual information that often help developers understand code better. For the experiments, we study code completion in two granularity levels; token and line completion and take three recent and large-scale language models for source code: UniXcoder …
Extending Source Code Pre-trained Language Models to Summarise Decompiled Binaries

Ali Al-Kaswan, Toufique Ahmed, Maliheh Izadi, Anand Ashok Sawant, Premkumar Devanbu, and Arie Deursen

In IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), 2023

Abs

Binary reverse engineering is used to understand and analyse programs for which the source code is unavailable. Decompilers can help, transforming opaque binaries into a more readable source code-like representation. Still, reverse engineering is difficult and costly, involving considering effort in labelling code with helpful summaries. While the automated summarisation of decompiled code can help reverse engineers understand and analyse binaries, current work mainly focuses on summarising source code, and no suitable dataset exists for this task. In this work, we extend large pre-trained language models of source code to summarise de-compiled binary functions. Further-more, we investigate the impact of input and data properties on the performance of such models. Our approach consists of two main components; the data and the model. We first build CAPYBARA, a dataset of 214K decompiled function …
Semantically-enhanced Topic Recommendation Systems for Software Projects

Maliheh Izadi, Mahtab Nejati, and Abbas Heydarnoori

Empirical Software Engineering, 2023

Abs

Software-related platforms such as GitHub and Stack Overflow, have enabled their users to collaboratively label software entities with a form of metadata called topics. Tagging software repositories with relevant topics can be exploited for facilitating various downstream tasks. For instance, a correct and complete set of topics assigned to a repository can increase its visibility. Consequently, this improves the outcome of tasks such as browsing, searching, navigation, and organization of repositories. Unfortunately, assigned topics are usually highly noisy, and some repositories do not have well-assigned topics. Thus, there have been efforts on recommending topics for software projects, however, the semantic relationships among these topics have not been exploited so far. In this work, we propose two recommender models for tagging software projects that incorporate the semantic relationship among topics. Our …
Stacc: Code comment Classification using SentenceTransformers

Ali Al-Kaswan, Maliheh Izadi, and Arie Van Deursen

In IEEE/ACM 2nd International Workshop on Natural Language-Based Software Engineering (NLBSE), 2023

Best Tool Award 🏆 Abs

We won the first place in the NLBSE competition.

Code comments are a key resource for information about software artefacts. Depending on the use case, only some types of comments are useful. Thus, automatic approaches to clas-sify these comments have been proposed. In this work, we address this need by proposing, STACC, a set of SentenceTransformers- based binary classifiers. These lightweight classifiers are trained and tested on the NLBSE Code Comment Classification tool competition dataset, and surpass the baseline by a significant margin, achieving an average Fl score of 0.74 against the baseline of 0.31, which is an improvement of 139%. A replication package, as well as the models themselves, are publicly available.
Targeted Attack on GPT-Neo for the SATML Language Model Data Extraction Challenge

Ali Al-Kaswan, Maliheh Izadi, and Arie Deursen

In The 1st IEEE Conference on Secure and Trustworthy Machine Learning, 2023

Best Technique Award 🏆 Abs

We received the first place in the SaTML competition.

Previous work has shown that Large Language Models are susceptible to so-called data extraction attacks. This allows an attacker to extract a sample that was contained in the training data, which has massive privacy implications. The construction of data extraction attacks is challenging, current attacks are quite inefficient, and there exists a significant gap in the extraction capabilities of untargeted attacks and memorization. Thus, targeted attacks are proposed, which identify if a given sample from the training data, is extractable from a model. In this work, we apply a targeted data extraction attack to the SATML2023 Language Model Training Data Extraction Challenge. We apply a two-step approach. In the first step, we maximise the recall of the model and are able to extract the suffix for 69% of the samples. In the second step, we use a classifier-based Membership Inference Attack on the generations. Our AutoSklearn classifier achieves a precision of 0.841. The full approach reaches a score of 0.405 recall at a 10% false positive rate, which is an improvement of 34% over the baseline of 0.301.
The (ab) Use of Open Source Code to Train Large Language Models

Ali Al-Kaswan, and Maliheh Izadi

In IEEE/ACM 2nd International Workshop on Natural Language-Based Software Engineering (NLBSE), 2023

Abs

In recent years, Large Language Models (LLMs) have gained significant popularity due to their ability to generate human-like text and their potential applications in various fields, such as Software Engineering. LLMs for Code are commonly trained on large unsanitized corpora of source code scraped from the Internet. The content of these datasets is memorized and emitted by the models, often in a verbatim manner. In this work, we will discuss the security, privacy, and licensing implications of memorization. We argue why the use of copyleft code to train LLMs is a legal and ethical dilemma. Finally, we provide four actionable recommendations to address this issue.

2022

CodeFill: Multi-token Code Completion by Jointly learning from Structure and Naming Sequences

Maliheh Izadi, Roberta Gismondi, and Georgios Gousios

In IEEE/ACM 44th International Conference on Software Engineering (ICSE), 2022

Abs HTML

Code completion is an essential feature of IDEs, yet current auto-completers are restricted to either grammar-based or NLP-based single token completions. Both approaches have significant drawbacks: grammar-based autocompletion is restricted in dynamically-typed language environments, whereas NLP-based autocompleters struggle to understand the semantics of the programming language and the developer’s code context.In this work, we present CodeFill, a language model for autocompletion that combines learned structure and naming information. Using a parallel Transformer architecture and multi-task learning, CodeFill consumes sequences of source code token names and their equivalent AST token types. Uniquely, CodeFill is trained both for single-token and multi-token (statement) prediction, which enables it to learn long-range dependencies among grammatical and naming elements. We train …
Predicting the Objective and Priority of Issue Reports in Software Repositories

Maliheh Izadi, Kiana Akbari, and Abbas Heydarnoori

Empirical Software Engineering, 2022

Abs

Software repositories such as GitHub host a large number of software entities. Developers collaboratively discuss, implement, use, and share these entities. Proper documentation plays an important role in successful software management and maintenance. Users exploit Issue Tracking Systems, a facility of software repositories, to keep track of issue reports, to manage the workload and processes, and finally, to document the highlight of their team’s effort. An issue report is a rich source of collaboratively-curated software knowledge, and can contain a reported problem, a request for new features, or merely a question about the software product. As the number of these issues increases, it becomes harder to manage them manually. GitHub provides labels for tagging issues, as a means of issue management. However, about half of the issues in GitHub’s top 1000 repositories do not have any labels. In this work, we aim at …
An Empirical Study on Data Leakage and Generalizability of Link Prediction Models for Issues and Commits

Maliheh Izadi, Pooya Rostami Mazrae, Tom Mens, and Arie Deursen

2022

Abs

To enhance documentation and maintenance practices, developers conventionally establish links between related software artifacts manually. Empirical research has revealed that developers frequently overlook this practice, resulting in significant information loss. To address this issue, automatic link recovery techniques have been proposed. However, these approaches primarily focused on improving prediction accuracy on randomly-split datasets, with limited attention given to the impact of data leakage and the generalizability of the predictive models. LinkFormer seeks to address these limitations. Our approach not only preserves and improves the accuracy of existing predictions but also enhances their alignment with real-world settings and their generalizability. First, to better utilize contextual information for prediction, we employ the Transformer architecture and fine-tune multiple pre-trained models on both textual and metadata information of issues and commits. Next, to gauge the effect of time on model performance, we employ two splitting policies during both the training and testing phases; randomly- and temporally-split datasets. Finally, in pursuit of a generic model that can demonstrate high performance across a range of projects, we undertake additional fine-tuning of LinkFormer within two distinct transfer-learning settings. Our findings support that to simulate real-world scenarios effectively, researchers must maintain the temporal flow of data when training models. Furthermore, the results demonstrate that LinkFormer outperforms existing methodologies by a significant margin, achieving a 48% improvement in F1-measure within a …
CatIss: An Intelligent Tool for Categorizing Issues Reports using Transformers

Maliheh Izadi

In IEEE/ACM 1st International Workshop on Natural Language-Based Software Engineering (NLBSE), 2022

Best Tool Award 🏆 Abs

I won the first place in the NLBSE competition.

Users use Issue Tracking Systems to keep track and manage issue reports in their repositories. An issue is a rich source of software information that contains different reports including a problem, a request for new features, or merely a question about the software product. As the number of these issues increases, it becomes harder to manage them manually. Thus, automatic approaches are proposed to help facilitate the management of issue reports. This paper describes CatIss, an automatic Categorizer of Issue reports which is built upon the Transformer-based pre-trained RoBERTa model. CatIss classifies issue reports into three main categories of Bug report, Enhancement/feature request, and Question. First, the datasets provided for the NLBSE tool competition are cleaned and preprocessed. Then, the pre-trained RoBERTa model is fine-tuned on the preprocessed dataset. Evaluating CatIss on about 80 …
On the Evaluation of NLP-based Models for Software Engineering

Maliheh Izadi, and Matin Nili Ahmadabadi

In IEEE/ACM 1st International Workshop on Natural Language-Based Software Engineering (NLBSE), 2022

Abs

NLP-based models have been increasingly incorporated to address SE problems. These models are either employed in the SE domain with little to no change, or they are greatly tailored to source code and its unique characteristics. Many of these approaches are considered to be outperforming or complementing existing solutions. However, an important question arises here: Are these models evaluated fairly and consistently in the SE community?. To answer this question, we reviewed how NLP-based models for SE problems are being evaluated by researchers. The findings indicate that currently there is no consistent and widely-accepted protocol for the evaluation of these models. While different aspects of the same task are being assessed in different studies, metrics are defined based on custom choices, rather than a system, and finally, answers are collected and interpreted case by case. Consequently, there …
Limits of Binary Code Summarization with Transformers

Ali Al-kaswan

2022

2021

Automated Recovery of Issue-commit Links Leveraging both Textual and Non-textual Data

Pooya Rostami Mazrae, Maliheh Izadi, and Abbas Heydarnoori

In IEEE International Conference on Software Maintenance and Evolution (ICSME), 2021

Abs

An issue report documents the discussions around required changes in issue-tracking systems, while a commit contains the change itself in the version control systems. Recovering links between issues and commits can facilitate many software evolution tasks such as bug localization, defect prediction, software quality measurement, and software documentation. A previous study on over half a million issues from GitHub reports only about 42.2% of issues are manually linked by developers to their pertinent commits. Automating the linking of commit-issue pairs can contribute to the improvement of the said tasks. By far, current state-of-the-art approaches for automated commit-issue linking suffer from low precision, leading to unreliable results, sometimes to the point that imposes human supervision on the predicted links. The low performance gets even more severe when there is a lack of textual information in either …
Topic Recommendation for Software Repositories using Multi-label Classification Algorithms

Maliheh Izadi, Abbas Heydarnoori, and Georgios Gousios

Empirical Software Engineering, 2021

Abs

Many platforms exploit collaborative tagging to provide their users with faster and more accurate results while searching or navigating. Tags can communicate different concepts such as the main features, technologies, functionality, and the goal of a software repository. Recently, GitHub has enabled users to annotate repositories with topic tags. It has also provided a set of featured topics, and their possible aliases, carefully curated with the help of the community. This creates the opportunity to use this initial seed of topics to automatically annotate all remaining repositories, by training models that recommend high-quality topic tags to developers. In this work, we study the application of multi-label classification techniques to predict software repositories’ topics. First, we map the large-space of user-defined topics to those featured by GitHub. The core idea is to derive more information from projects’ available …

2020

Generating Summaries for Methods of Event-driven Programs: An Android Case Study

Alireza Aghamohammadi, Maliheh Izadi, and Abbas Heydarnoori

Journal of Systems and Software, 2020

Abs

The lack of proper documentation makes program comprehension a cumbersome process for developers. Source code summarization is one of the existing solutions to this problem. Many approaches have been proposed to summarize source code in recent years. A prevalent weakness of these solutions is that they do not pay much attention to interactions among elements of software. An element is simply a callable code snippet such as a method or even a clickable button. As a result, these approaches cannot be applied to event-driven programs, such as Android applications, because they have specific features such as numerous interactions between their elements. To tackle this problem, we propose a novel approach based on deep neural networks and dynamic call graphs to generate summaries for methods of event-driven programs. First, we collect a set of comment/code pairs from Github and train a deep …
Improving Quality of a Post’s Set of Answers in Stack Overflow

Mohammadreza Tavakoli, Maliheh Izadi, and Abbas Heydarnoori

In 46th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), 2020

Abs

Community Question Answering platforms such as Stack Overflow help a wide range of users solve their challenges on-line. As the popularity of these communities has grown over the years, both the number of members and posts have escalated. Also, due to the diverse backgrounds, skills, expertise, and viewpoints of users, each question may obtain more than one answer. Therefore, the focus has changed toward producing posts that have a set of answers more valuable for the community as a whole, not just one accepted-answer aimed at satisfying only the question-asker. Same as every universal community, a large number of low-quality posts on Stack Overflow require improvement. We call these posts "deficient", and define them as posts with questions that either have no answer yet or can be improved by other ones. In this paper, we propose an approach to automate the identification process of such posts …

2018

Evaluating Collaborative Filtering Recommender Algorithms: a Survey

Mahdi Jalili, Sajad Ahmadian, Maliheh Izadi, Parham Moradi, and Mostafa Salehi

IEEE access, 2018

Abs

Due to the explosion of available information on the Internet, the need for effective means of accessing and processing them has become vital for everyone. Recommender systems have been developed to help users to find what they may be interested in and business owners to sell their products more efficiently. They have found much attention in both academia and industry. A recommender algorithm takes into account user–item interactions, i.e., rating (or purchase) history of users on items, and their contextual information, if available. It then provides a list of potential items for each target user, such that the user is likely to positively rate (or purchase) them. In this paper, we review evaluation metrics used to assess performance of recommendation algorithms. We also survey a number of classical and modern recommendation algorithms and compare their performance in terms of different evaluation metrics on …

2017

The Intonation Patterns of English and Persian Sentences: A Contrastive Study

Mehri Izadi, Maliheh Izadi, and Behnam Azarsa

Research Journal of Education, 2017

Abs

Different intonation pattern is one of the factors affecting the learning of L2 pronunciation. The contrastive analysis of English-Persian intonation patterns has shown that both languages are similar in sentence-final intonation while they are different in incomplete sentences. To this end, this paper describes English-Persian intonation patterns to look at the differences and similarities of the two languages to improve the effectiveness of L2 learning.Â

2015

Recommender Systems for Social Networks Analysis and Mining: Precision versus Diversity

Amin Javari, Maliheh Izadi, and Mahdi Jalili

2015

Abs

Recommender systems has become increasingly important in online community for providing personalized services and products to users. Traditionally, performance of recommender algorithms has been evaluated based on accuracy and the focus of the research was on providing accurate recommendation lists. However, recently diversity and novelty of recommendation lists have been introduced as key issues in designing recommender systems. In general, novelty/diversity and accuracy do not go hand in hand. Therefore, designing models answering novelty/diversity-accuracy dilemma is one of the challenging problems in the context of practical recommender systems. In this paper, we first introduce the diversity-accuracy challenge in recommender systems, and then present two recommendation algorithms which approach the problem from two perspectives. The first model is a filtering algorithm to …

2014

Unifying Inconsistent Evaluation Metrics in Recommender Systems

Maliheh Izadi, Amin Javari, and Mahdi Jalili

REDD Workshop, co-located with RecSys, 2014

Abs

Recommender systems are among the most popular tools used by online community these days. Traditionally, recommender techniques were evaluated using accuracy-based metrics such as precision; however, gradually the need for other qualities including more novel and diverse items emerged. Consequently, researchers started to evaluate their finding with different and often inconsistent metrics and made it nearly impossible to compare the existing approaches properly. It is clear that we need a more unified approach to assess the results of new techniques, and to the best of our knowledge, this problem has not been answered yet in previous studies. In this paper, we proposed a novel and extensible framework for evaluation of recommender systems using maximum bounds of possible measures in different datasets. Finally we provided the results of applying this framework on a set of different recommender algorithms.