My research focuses on improving the quality and correctness of code generated by Large Language Models (LLM) aiming to improve their reliability for software engineering and reasoning tasks. Most recently, I have been focusing on repository-level reasoning for code generation with LLMs. I have also explored long context usage, tool usage, better tokenization, prompting for code and decoding techniques with LLMs. Previously, I have worked in Programming Languages and Systems having developed language runtimes, IDE/Debugger support for languages, and source-to-source transpilers.
↴ (click to expand details on my projects)
Keywords: Large Language Models, AI4Code, Code Generation, Static Analysis, Software Engineering, LLM Tool Usage, LLM Decoding Techniques
For more details about my background, refer to my CV. If you'd like to chat with me about my work or research in general, feel free to reach out via email! If you would like to contact me anonymously, kindly fill this anonymous form.
Monitor-Guided Decoding of Code LMs with Static Analysis of Repository Context
, Aditya Kanade, Navin Goyal, Shuvendu Lahiri, Sriram Rajamani
Neural Information Processing Systems
(NeurIPS), 2023
PDF Poster Slides NeurIPS Page with Virtual Talk OpenReviewGitHub
A SPARQL to Cypher Transpiler: Proposal and Initial Results. Extended Abstract
, Nikunj Singhal, Raghava Mutharaju
International Conference on Data Science and Management of Data (CODS-COMAD), 2022
PDF ACM DL Page Google Scholar
A SPARQL to cypher transpiler
, Nikunj Singhal, Raghava Mutharaju
Undergraduate Thesis, Computer Science and Applied Mathematics, IIIT-Delhi, 2021 - 22
PDF Poster IIIT-D Archive Google Scholar
A novel sentiment analysis engine for preliminary depression status estimation on social media
Sudhir Kumar Suman, Hrithwik Shalu,
, Archit Agrawal, Juned Kadiwala
Preprint arXiv:2011.14280, 2020
PDF Google Scholar
- Lead the work on "Monitor-Guided Decoding of Code LMs with Static Analysis of Repository Context" (OpenReview, GitHub) which proposes Monitor Guided Decoding (MGD), a novel decoding technique combining dynamic constrained decoding with language-server-protocol (LSP) based external tool usage. MGD was accepted at NeurIPS '23, and also won first place in Microsoft Global Hackathon on improving productivity. MGD can prevent hallucinated symbols and methods, ensure methods are called in correct order at runtime (following a typestate specification), and have correct number of arguments to function calls, thus preventing various compilation, runtime and security errors in LLM generated code at a minimal overhead. With MGD, we show that even small LMs of size 350M can achieve better compilation rate and ground truth match than much larger LM of size 175B, and achieve 20-25% improvements in compilation rate for generated code across all model sizes from 350M-175B.
- Developed multilspy, an OSS library to easily use and launch different language servers, easing the process of creating language server clients for various applications including AI for Code scenarios. Language servers are tools that perform a variety of static analyses on source code and provide useful information such as type-directed code completion suggestions, symbol definition locations, symbol references, etc. multilspy abstracts the setting up of the language servers, performs language-specific configuration and handles communication with the server over the json-rpc based protocol, while exposing a simple interface to the user, allowing LSP use in just 3 lines of code!
- Curated PragmaticCode and DotPrompts, large benchmark of buildable java repositories, which provides a unified harness to compile a diverse set of Java projects abstracting multiple build systems, thus allowing for pragmatic evaluations of Code-LMs. The datasets consist of 10,000+ prompts that require repository-level understanding to complete. Each prompt is a method-completion task.
- As a research intern under Prof. James Larus at VLSC Lab, EPFL, I developed StreamBlocks GraalVM, the CPU runtime for CAL (a dataflow programming language), along with IDE and debugger support for it (Google Slides, GitHub).
- As a Google Summer of Code student, I developed Pytranslate, a programming language transpiler to convert Maxima (computer algebra system) to Python, implemented in Common Lisp. It is now a part of all Maxima installations.
- I am passionate for open source software, and apart from my own open source projects, I have contributed to projects like INRIA/spoon, lucidrains/memorizing-transformers-pytorch, mozilla/bugbug. You can find more about my open source work on GitHub: LakshyAAAgrawal.
Venues: Microsoft Research RiSE Group, Microsoft Research India, Microsoft DevDiv (July, August 2023)
abstract slides
Venues: Very Large Scale Computing Lab, Data Center Systems Laboratory (September 2021)
GitHub slides
Presenting MGD at NeurIPS '23
Receiving certificate from the Chief Minister of Chhattisgarh
Presenting MGD at NeurIPS '23
Receiving Institute Medals at IIIT-D Convocation
Presenting MGD at Microsoft Global Hackathon
Website source and acknowledgements