AI

• Adam: A Method for Stochastic Optimization

  Diederik P. Kingma and Jimmy Ba · Report

  The 2015 paper introducing the Adam optimizer, combining adaptive per-parameter learning rates with momentum. Became the default optimizer for deep learning and remains the standard choice for training large models.

  Also filed under Machine Learning, Math

• Agile Machine Learning: Effective Machine Learning Inspired by the Agile Manifesto

  Eric Carter & Matthew Hurst · Book

  Trademarked names, logos, and images may appear in this book. Rather than use a trademark symbol with every occurrence of a trademarked name, logo, or image we use the names, logos, and images only in an editorial fashion and to the benefit of the trademark owner, with no intention of infringement of the trademark. The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not identified as such, is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights. While the advice and information in this ...

  Also filed under Machine Learning, Math

• Artificial Intelligence Index Report 2024

  Ray Perrault & Jack Clark · Book

  The "Artificial Intelligence Index Report 2024" is the seventh edition of an annual comprehensive analysis that tracks, collates, distills, and visualizes data related to artificial intelligence (AI). Produced by the Stanford Institute for Human-Centered Artificial Intelligence (HAI), this report aims to provide unbiased, rigorously vetted, and broadly sourced data to inform policymakers, researchers, executives, journalists, and the general public about the complex field of AI. Key Highlights: Expansion of Scope: This edition broadens its coverage to include essential trends such as technical advancements in AI, public perceptions of the technology, and the geopolitical dynamics surrounding its development. Original Data and New Analyses: Featuring more original data than previous editions, the report introduces new estimates on AI training costs, detailed analyses of the responsible AI landscape, and an entirely new chapter dedicated to AI's impact on science and medicine. Generative AI Investment Surge: Despite a decline in overall AI private investment in 2023, funding for generative AI surged, nearly octupling from 2022 to reach $25.2 billion. Major players in the generative AI space, including OpenAI, Anthropic, Hugging Face, and Inflection, reported substantial fundraising rounds. U.S. Leadership in AI Investment: The United States saw AI investments reach $67.2 billion in 2023, nearly 8.7 times more than China, the next highest country. Impact on Science and Medicine: An entirely new chapter is dedicated to AI's impact on science and medicine, reflecting the growing influence of AI technologies in these critical fields. This report serves as an essential resource for understanding the current state and future trajectory of AI, offering a blend of data-driven insights and comprehensive analyses across various dimensions of artificial intelligence.

  Also filed under Machine Learning, Statistics

• Artificial Intelligence with Python

  Tutorials Point · Book

  About the Tutorial Artificial intelligence is the intelligence demonstrated by machines, in contrast to the intelligence displayed by humans. This tutorial covers the basic concepts of various fields of artificial intelligence like Artificial Neural Networks, Natural Language Processing, Machine Learning, Deep Learning, Genetic algorithms etc., and its implementation in Python. Audience This tutorial will be useful for graduates, post graduates, and research students who either have an interest in this subject or have this subject as a part of their curriculum. The reader can be a beginner ...

  Also filed under Python, Machine Learning

• Artificial Intelligence/ Machine Learning Explained

  Steve Blank · Guide

  about the competitive edge they’d have by today in business or as a nation. That’s where we are today with Artificial Intelligence and Machine Learning. These technologies will transform businesses and government agencies. Today, 100s of billions of dollars in private capital have been invested in 1,000s of AI startups.

  Also filed under Machine Learning, Statistics

• Attention Is All You Need

  Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin (Google Brain) · Report

  The 2017 NeurIPS paper that introduced the Transformer architecture — the foundation of every modern large language model. Dispensing with recurrence and convolutions entirely, the Transformer uses multi-head self-attention to parallelize sequence modeling. The single most-cited deep learning paper of the last decade.

  Also filed under Machine Learning, Computer Science

• Automated Machine Learning: Methods, Systems, Challenges

  Frank Hutter, Lars Kotthoff, Joaquin Vanschoren (Editors) · Book

  An edited volume in The Springer Series on Challenges in Machine Learning, covering methods, systems, and open challenges in automated machine learning (AutoML). Contributions span algorithmic foundations, hyperparameter optimization, neural architecture search, and real-world applications.

  Also filed under Machine Learning, Computer Science

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

  Sergey Ioffe and Christian Szegedy (Google) · Report

  The 2015 paper introducing batch normalization — a technique that normalizes layer inputs during training, dramatically accelerating convergence and enabling higher learning rates. Standard component of modern CNN and Transformer architectures.

  Also filed under Machine Learning, Statistics

• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

  Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova (Google AI Language) · Report

  The 2018 paper introducing BERT, which established the pre-train/fine-tune paradigm that dominated NLP before the GPT era. Masked language modeling + next-sentence prediction on unlabeled text produced representations transferable to a wide range of downstream tasks.

  Also filed under Machine Learning, Computer Science

• Big Data and Artificial Intelligence in Digital Finance

  John Soldatos & Dimosthenis Kyriazis · Book

  Preface The finance sector is among the most data-savvy and data-intensive of the global economy. The ongoing digital transformation of financial organizations, along with their interconnection as part of a global digital finance ecosystem, is producing petabytes of structured and unstructured data. The latter represent a significant opportunity for banks, financial institutions, and financial technology firms (Fin- Techs): Leveraging these data financial organizations can significantly improve both their business processes and the quality of their decisions. As a prominent example, modern banks can...

  Also filed under Quant Finance, Machine Learning

• Building Machine Learning Systems with a Feature Store

  Jim Dowling · Book

  O'Reilly Media early-release (2025) on designing and operating ML systems built around a feature store. Covers feature engineering pipelines, online/offline serving, data consistency, and the architecture of modern ML platforms.

  Also filed under Machine Learning, Software Engineering

• Constitutional AI: Harmlessness from AI Feedback

  Yuntao Bai et al. (Anthropic) · Report

  The 2022 Constitutional AI paper introducing RLAIF — reinforcement learning from AI feedback — as a scalable alternative to RLHF. Models critique and revise their own outputs against a set of written principles. The foundation of Claude's training methodology.

  Also filed under Machine Learning

• Deep Learning with Azure

  Mathew Salvaris, Danielle Dean, Wee Hyong Tok · Book

  "Deep Learning with Azure" by Mathew Salvaris, Danielle Dean, and Wee Hyong Tok is a practical guide for data scientists, engineers, and AI practitioners who want to harness the power of Microsoft Azure to build, train, and deploy deep learning models. Through a series of hands-on tutorials, the book covers everything from setting up the development environment and creating deep learning models to deploying those models at scale on Azure. Key Concepts Covered Introduction to Azure and Deep Learning: The book begins with an introduction to Azure, particularly Azure Machine Learning (AML), a cloud-based service that simplifies the deployment of machine learning models. It explains how to use Azure's services to handle the computational resources required for deep learning tasks. Setting Up Your Deep Learning Environment: A key focus of the book is the setup and configuration of the deep learning environment on Azure. The authors walk through the steps required to set up Azure Machine Learning workspaces, create compute clusters, and use Azure Notebooks for efficient experimentation. The authors also discuss the integration of Azure Databricks for big data analytics and model training. Building Deep Learning Models: The book provides practical, hands-on examples of building various types of deep learning models, including convolutional neural networks (CNNs), recurrent neural networks (RNNs), and generative models, using frameworks like TensorFlow, Keras, and PyTorch on Azure. By the end of the book, readers will have the skills necessary to leverage Azure’s cloud capabilities for deep learning, optimizing workflows, scaling model training, and deploying models for real-world applications. The book provides a comprehensive and structured approach to using Azure for deep learning, making it an essential resource for anyone working with AI in the cloud.

  Also filed under Machine Learning, SQL

• Deep Neural Networks and Data for Automated Driving

  Tim Fingscheidt, Hanno Gottschalk & Sebastian Houben · Book

  This book is an open access publication. The images or other third party material in this book are included in the book’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the book’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. The use of general descriptive names, registered names, trademarks, service marks, etc.

  Also filed under Machine Learning, Computer Science

• Deep Residual Learning for Image Recognition

  Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun (Microsoft Research) · Report

  The 2015 paper introducing residual connections and ResNet — the architecture that enabled training of networks 100+ layers deep. Residual connections solved the degradation problem in deep networks and became a near-universal building block of modern deep learning.

  Also filed under Machine Learning, Computer Science

• Dropout: A Simple Way to Prevent Neural Networks from Overfitting

  Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov (University of Toronto) · Report

  The 2014 JMLR paper introducing dropout as a regularization technique. Randomly dropping units during training prevents co-adaptation and acts as an approximate ensemble method. A foundational technique used in nearly every deep neural network.

  Also filed under Machine Learning, Statistics

• Efficient Exploration for LLMs

  Vikranth Dwaracherla, Seyed Mohammad Asghari, Botao Hao, Benjamin Van Roy · Report

  "Efficient Exploration for LLMs" is a research paper authored by Vikranth Dwaracherla, Seyed Mohammad Asghari, Botao Hao, and Benjamin Van Roy, focusing on enhancing large language models (LLMs) through efficient exploration strategies in gathering human feedback. Key Contributions: Efficient Query Generation: The study introduces an agent that sequentially generates queries while concurrently fitting a reward model based on the feedback received. This approach aims to optimize the learning process by selecting the most informative queries. Double Thompson Sampling: The researchers employ double Thompson sampling for query generation, utilizing epistemic neural networks to represent uncertainty. This method balances exploration and exploitation, leading to more effective learning with fewer queries. Performance Improvement: The findings demonstrate that efficient exploration enables high levels of performance with significantly fewer queries, highlighting the importance of uncertainty estimation and the choice of exploration strategy in training LLMs. This research underscores the potential of incorporating advanced exploration techniques to enhance the efficiency and effectiveness of large language models, particularly in the context of human-in-the-loop training scenarios.

  Also filed under Machine Learning, Quant Finance

• Exploring Synthetic Data for Artificial Intelligence and Autonomous Systems: A Primer

  Harry Deng · Report

  UNIDIR (United Nations Institute for Disarmament Research) primer on synthetic data: how it is generated, why it is used in AI and autonomous systems, and the security and policy implications for militarized applications. Published 2023 under UNIDIR Security and Technology Programme.

  Also filed under Machine Learning

• Flexible Automation and Intelligent Manufacturing: The Human-Data- Technology Nexus

  Kyoung-Yun Kim, Leslie Monplaisir & Jeremy Rickli · Book

  This book is an open access publication, corrected publication 2023. The images or other third party material in this book are included in the book's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the book'

  Also filed under Machine Learning, SQL

• Foundation Models for Natural Language Processing

  Gerhard Paaß & Sven Giesselbach · Book

  It brings together the latest developments in all areas of this multidisciplinary topic, ranging from theories and algorithms to various important applications. Furthermore, it supports Open Access publica- tion mode. This book is an open access publication. The images or other third party material in this book are included in the book’s Creative Commons license, unless indicated otherwise in a credit line to the material.

  Also filed under Machine Learning, Software Engineering

• Generative Adversarial Nets

  Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio · Report

  The 2014 paper that introduced GANs — a framework where a generator and discriminator are trained in a minimax game. Opened the era of modern generative modeling and led directly to the image synthesis breakthroughs of the late 2010s.

  Also filed under Machine Learning, Statistics

• ImageNet Classification with Deep Convolutional Neural Networks

  Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton (University of Toronto) · Report

  The 2012 AlexNet paper that catalyzed the deep learning revolution. A deep CNN trained on two GPUs reduced the ImageNet top-5 error rate from 26% to 15%, convincing the broader ML community that deep networks were production-ready.

  Also filed under Machine Learning, Computer Science

• Language Models are Few-Shot Learners (GPT-3)

  Tom B. Brown et al. (OpenAI) · Report

  The 2020 GPT-3 paper demonstrating that scale alone — 175 billion parameters — unlocks few-shot in-context learning. The paper that transformed the commercial viability of large language models and kicked off the modern LLM era.

  Also filed under Machine Learning

• Lecture Notes On Artificial Intelligence

  Prashanta Kumar Patra · Book

  "Lecture Notes on Artificial Intelligence" is a structured and concise guide designed to introduce the fundamental concepts of Artificial Intelligence (AI). The book is tailored for students, researchers, and professionals looking to understand the theoretical foundations and practical applications of AI. It provides a systematic overview of AI methodologies, techniques, and algorithms, making it an essential resource for academic learning and real-world problem-solving. Key Topics Covered: Introduction to Artificial Intelligence: Definition, history, and evolution of AI, along with its impact on various industries. Problem Solving & Search Algorithms: Covers uninformed search (BFS, DFS), heuristic search (A), and optimization techniques*. Knowledge Representation & Reasoning: Explores logical reasoning, semantic networks, ontologies, and rule-based systems. Machine Learning Fundamentals: Introduces supervised, unsupervised, and reinforcement learning, along with key algorithms like decision trees and neural networks. Natural Language Processing (NLP): Discusses text processing, sentiment analysis, and AI-driven language models. Expert Systems & Fuzzy Logic: Explains the design and implementation of expert systems and how fuzzy logic enhances decision-making. Neural Networks & Deep Learning: Covers perceptrons, backpropagation, convolutional neural networks (CNNs), and recurrent neural networks (RNNs). AI Ethics & Future Trends: Discusses ethical considerations, AI governance, and the future impact of AI on society. With clear explanations, mathematical foundations, and practical examples, this book serves as a valuable resource for understanding AI principles and preparing for advanced studies or careers in artificial intelligence.

  Also filed under Machine Learning, Computer Science

• LLaMA: Open and Efficient Foundation Language Models

  Hugo Touvron et al. (Meta AI) · Report

  The 2023 paper releasing LLaMA — a family of 7B–65B parameter models trained on publicly available data. LLaMA-13B outperformed GPT-3 (175B) on most benchmarks. Opened the era of capable open-weight LLMs and enabled the explosion of open-source LLM work.

  Also filed under Machine Learning

• Representation Learning for Natural Language Processing

  Zhiyuan Liu, Yankai Lin & Maosong Sun · Book

  Preface In conventional natural language processing (NLP) systems, language items such as words and phrases are handled as distinct symbols. Many classical methods, such as n-gram and bag-of-words models, were proposed and have been widely used until now. All these methods take words as the minimum units for semantic representation, either used to estimate the conditional probabilities of the next word given previous words (e.g., n-gram) or used to represent semantic meanings of text (e.g., bag-of-words models). Even when people find it necessary to model word meanings, they either manually ...

  Also filed under Machine Learning, Data Visualization

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

  Patrick Lewis et al. (Facebook AI Research) · Report

  The 2020 paper that formalized RAG — combining a parametric seq2seq model with a non-parametric retrieval component. The architecture underlying most production LLM deployments that need current or private knowledge.

  Also filed under Machine Learning

• Scaling Laws for Neural Language Models

  Jared Kaplan et al. (OpenAI) · Report

  The 2020 Kaplan paper establishing empirical scaling laws for language models: loss is predictable as a power law in model size, dataset size, and compute. The paper that convinced OpenAI to bet on scale and led directly to GPT-3.

  Also filed under Machine Learning, Statistics

• The legal implications of Generative AI

  Donna Bartlett, Willem-Jan Cosemans, Matt Saunders, Till Contzen, Klaus Gresbrand, Maria-Alexandra Papoutsi, Pietro Boccaccini, Peggy Anstett, Bruce Braude, Richard Reeve-Young · Report

  The book explores the legal challenges and considerations surrounding the use of generative AI technologies. It examines issues such as intellectual property rights, data privacy, liability, and regulatory frameworks. The authors provide insights into how these technologies impact various industries and offer guidance on navigating the evolving legal landscape to ensure compliance and mitigate risks.

  Also filed under Machine Learning, Math

• Toolformer: Language Models Can Teach Themselves to Use Tools

  Timo Schick et al. (Meta AI) · Report

  The 2023 Toolformer paper showing how LLMs can be self-taught to invoke external APIs (calculator, search, translation) by inserting API call annotations into training data. A foundational paper in the agentic-LLM lineage.

  Also filed under Machine Learning

• Topologies of Reasoning: Demystifying Chains, Trees, and Graphs of Thoughts

  Maciej Besta, Florim Memedi, Zhenyu Zhang, Robert Gerstenberger, Nils Blach, Piotr Nyczyk, Marcin Copik, Grzegorz Kwasniewski, Jurgen Muller, Lukas Gianinazzi, Ales Kubicek, Hubert Niewiadomski, Onur Mutlu,Torsten Hoefler · Report

  "Topologies of Reasoning: Demystifying Chains, Trees, and Graphs of Thoughts" is a comprehensive study by Maciej Besta, Florim Memedi, Zhenyu Zhang, Robert Gerstenberger, Nils Blach, Piotr Nyczyk, Marcin Copik, Grzegorz Kwaśniewski, Jürgen Müller, Lukas Gianinazzi, Ales Kubicek, Hubert Niewiadomski, Onur Mutlu, and Torsten Hoefler, published in January 2024. The paper explores structured prompting techniques in large language models (LLMs), focusing on how different reasoning topologies—chains, trees, and graphs—can enhance model performance across various tasks. Key Highlights: Structured Prompting Techniques: The study introduces paradigms like Chain-of-Thought, Tree of Thoughts, and Graph of Thoughts, which guide LLM reasoning through explicit structures, leading to improved task-solving capabilities. Taxonomy of Reasoning Topologies: The authors present a taxonomy categorizing structure-enhanced LLM reasoning schemes, analyzing aspects such as topology class, scope, representation, and execution algorithms. Comparative Analysis: By dissecting existing prompting schemes, the paper evaluates how design choices influence performance metrics like accuracy, latency, and cost, providing insights into effective prompting strategies. Theoretical Insights and Future Challenges: The paper discusses the interplay between prompting techniques and other LLM components, highlighting research challenges and suggesting directions for future advancements in prompt engineering. This work serves as a valuable resource for understanding and advancing structured prompting methods in LLMs, offering a foundation for developing more efficient and effective language models.

  Also filed under Machine Learning, Math

• Training Compute-Optimal Large Language Models (Chinchilla)

  Jordan Hoffmann et al. (DeepMind) · Report

  The 2022 Chinchilla paper that corrected Kaplan's scaling laws. Trained 400+ models from 70M to 16B parameters, showing that model size and training tokens should scale equally — a 70B Chinchilla with 4× more data beat 280B Gopher. Reshaped the training strategy for every subsequent frontier LLM.

  Also filed under Machine Learning, Statistics

• Training Language Models to Follow Instructions with Human Feedback (InstructGPT)

  Long Ouyang et al. (OpenAI) · Report

  The 2022 InstructGPT paper establishing the RLHF recipe: supervised fine-tuning on high-quality demonstrations, reward model training from human preferences, and PPO optimization. The 1.3B InstructGPT beat the 175B GPT-3 on human preference — proof that alignment mattered more than scale.

  Also filed under Machine Learning

• Unlocking Artificial Intelligence

  Christopher Mutschler, Christian Münzenmayer, Norman Uhlmann & Alexander Martin · Book

  This book is an open access publication. The images or other third party material in this book are included in the book’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the book’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. The use of general descriptive names, registered names, trademarks, service marks, etc.

  Also filed under Machine Learning, Math

The documents are the work of their respective authors and publishers; Gyre Research claims no ownership and will remove any document on request from a rightsholder — team@gyreresearch.com. The summaries and subject classifications are original work by Gyre Research and may be quoted with attribution.