Papers I plan to read.

Surveys

• State of AI 2024, Stanford

LLMs training / inference / architecture

• Attention Is All You Need
• Mamba
• Jamba
• xLSTM
• KAN
• LSTM
• EFFICIENT STREAMING LANGUAGE MODELS WITH ATTENTION SINKS
• Hidden Attention of Mamba Models
• Mixture Of Depth

Increasing context length

• MEGALODON: Efficient LLM Pretraining and Inference with Unlimited Context Length
• EFFICIENT STREAMING LANGUAGE MODELS WITH ATTENTION SINKS
• Leave No Context Behind

Interpret-ability

• Physics of Language Models

Tool integration w/ LLMs

• Gorilla: Large Language Model Connected with Massive APIs Patil et al. (2023)
• MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action Yang et al. (2023)
• Efficient Tool Use with Chain-of-Abstraction Reasoning Gao et al. (2024)
• THE LANDSCAPE OF EMERGING AI AGENT ARCHITECTURES FOR REASONING, PLANNING, AND TOOL CALLING: A SURVEY

In-Context Learning

• Reddit discussion
  • Another
• What Makes In-Context Learning Work?
• Mind the instructions
• Learning Algorithm, You Tube
• Many shot in-context learning
• Transformers Learn Higher-Order Optimization Methods for In-Context Learning
• Anthropic: In-context Learning and Induction Heads
• In-Context Learning Creates Task Vectors
• A mechanistic understanding of chain-of-thought reasoning
• Intention
• Transformer Circuits, Induction Heads, In-Context Learning,, Stanford, CS25
• Transformers Learn In-Context by Gradient Descent

Planning with AI

• Searchformer

Quantization

• int8
• QLoRA
• AWQ
• GPTQ
• Smooth Quant
• QuIP
• HQQ
• AQLM

Self-Supervised Learnings

• pretext-based
  • UNSUPERVISED REPRESENTATION LEARNING BY PRE- DICTING IMAGE ROTATIONS
  • Solving Jigsaw Puzzles
• contrastive-based
  • Contrastive Learning of Visual Representations
  • Momentum Contrast for Unsupervised Visual Representation Learning
  • Improved Baselines with Momentum Contrastive Learn
• Architectural change
  • Bootstrap Your Own Latent A New Approach to Self-Supervised Learning
  • Exploring Simple Siamese Representation Learning
• Feature de-correlation-base
  • VICREG
  • Barlow Twins

Closer to Metal

• Sharding
  • megatron 1 (tp)
  • megatron 2 (pp)
  • megatron 3 (sequence/selective)
  • tp+fsdp w/ jax
  • Techniques-for-training-large-neural-networks
  • GSPMD partitioner
  • GShard
  • MoE:
    • switch
    • base