EADST

Review: H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models

Title: [NeurIPS'23] H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models.

Rating: Average, Not Recommended

Paper:H2O Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models

Code:https://github.com/FMInference/H2O

Review:

The paper introduces an approach to reduce KV cache in large language models by selecting the top k values as the main KV Cache for the following computing. However, the presentation of Algorithm 1 is complex and lacks clarity, for example, the element 'Gi' not being mentioned in the paper. This obscurity makes it difficult to understand the main idea. Further, the actual method, as revealed by the code, is disappointing, at least for LLAMA. It merely involves applying softmax to the QK matrix for updated scores and then selecting the top k indices for the mask update. This approach seems overly simplistic and does not match the expected sophistication of the proposed solution.

该论文提出了一种减少大型语言模型中KV缓存的方法,通过选择计算中的前k个数值作为主要KV缓存进行后续的计算。然而,算法1的呈现复杂且缺乏清晰度,比如元素'Gi'未提及,使得算法难以理解。进一步查看其代码,实际方法令人失望,至少对于LLAMA模型应用来讲。它只是对QK矩阵应用softmax得到更新的分数,然后选择前k个索引更新掩码。这种方法过于简单,并未达到预期的复杂性和提出的解决方案的深度。

modify_llama.py def local_heavy_hitter_mask

def local_heavy_hitter_mask(attn_weights, heavy_budget):
    # 获取注意力权重的数据类型和序列长度
    dtype_attn_weights = attn_weights.dtype
    seq_length = attn_weights.shape[-1]
    padding_length = 0

    # 对注意力权重应用softmax函数以获得正规化的注意力分布
    tmp_attn = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(dtype_attn_weights)

    # 计算前heavy_budget个令牌的累积注意力分数
    accumulated_attention_score = torch.sum(tmp_attn[:,:,padding_length:heavy_budget+padding_length,:], dim=-2) # (head, keys)
    # 对超出预算的部分分数置零
    accumulated_attention_score[:,:,heavy_budget+padding_length:] = 0
    accumulated_attention_score[:,:,:padding_length] = 0

    # 初始化一个与注意力权重形状相同的零掩码张量
    mask_bottom = torch.zeros_like(attn_weights, dtype=torch.bool)
    # 在掩码中标记重击者区域为True
    mask_bottom[:,:, padding_length:heavy_budget+padding_length, padding_length:heavy_budget+padding_length] = True

    # 遍历序列中的每个令牌
    for token_index in range(heavy_budget+padding_length, seq_length):
        # 计算当前令牌的softmax注意力权重
        tmp_attn_index = nn.functional.softmax(attn_weights[:,:,token_index,:], dim=-1, dtype=torch.float32).to(dtype_attn_weights)
        # 选择前heavy_budget-1个最高的累积注意力分数
        _, tmp_topk_index = accumulated_attention_score.topk(k=heavy_budget-1, dim=-1)
        # 创建当前令牌的零掩码张量
        zeros_index = torch.zeros_like(tmp_attn_index, dtype=torch.bool)
        # 更新当前令牌的掩码,标记最高分数的位置
        mask_bottom_index = zeros_index.scatter(-1, tmp_topk_index, True) # (head, keys)
        mask_bottom_index[:,:, token_index] = True

        # 更新总掩码,将当前令牌的掩码信息添加进去
        mask_bottom[:,:,token_index,:] = mask_bottom_index
        # 更新累积注意力分数
        accumulated_attention_score += tmp_attn_index
        accumulated_attention_score = accumulated_attention_score * mask_bottom_index

    # 返回计算得到的掩码
    return mask_bottom
相关标签
About Me
XD
Goals determine what you are going to be.
Category
标签云
阿里云 Quantize Math icon Github VSCode FP8 Algorithm Python llama.cpp Hilton ResNet-50 Qwen 签证 RL FlashAttention OpenCV OpenAI scipy 顶会 Tensor Nginx Jupyter diffusers Ptyhon Baidu EXCEL 音频 Miniforge Gemma LLM WebCrawler AI Disk Mixtral 飞书 SPIE CSV Quantization Vim Michelin Jetson 图形思考法 FP64 Git TensorRT 搞笑 NLTK CTC 公式 Bin GPT4 Augmentation 强化学习 Paper Review WAN Use Web hf Bitcoin Color BeautifulSoup NameSilo Zip mmap Input Template ms-swift Bert SAM PDF TTS 长期记忆 Django 报税 torchinfo Random Conda LeetCode uWSGI Pillow Shortcut Image2Text Pytorch Diagram JSON OCR Llama PDB 个人博客 VGG-16 FastAPI BTC Attention LLAMA TSV Crawler Base64 DeepStream IndexTTS2 GoogLeNet tqdm PyCharm printf 版权 Password ONNX Clash GPTQ Numpy Qwen2 Plotly Linux CUDA BF16 Hungarian Translation v0.dev Proxy CEIR RGB QWEN Paddle Streamlit MD5 Vmess C++ Heatmap Tiktoken Card Tracking CLAP 多线程 腾讯云 TensorFlow Pickle Excel Statistics DeepSeek Dataset git YOLO Video Safetensors LaTeX Magnet Domain 图标 多进程 Pandas Cloudreve Hotel CV 算法题 Data uwsgi Datetime v2ray HaggingFace RAR 云服务器 LoRA Google ModelScope SQL SQLite COCO XML Ubuntu Markdown ChatGPT git-lfs Freesound Breakpoint Firewall Rebuttal 论文速读 tar SVR Harness API网关 GGML 继承 Jev News Plate Agent API FP16 Bipartite NLP CC VPN GIT 关于博主 transformers Distillation Docker 论文 Land 证件照 域名 UI Interview CAM Windows Anaconda Claude 递归学习法 第一性原理 净利润 自我改进 PIP Sklearn Logo FP32 HuggingFace UNIX Animate InvalidArgumentError Search Website Qwen2.5 PyTorch XGBoost logger 财报 Food Knowledge Permission Transformers
站点统计

本站现有博文340篇,共被浏览1018377次

本站已经建立2718天!

热门文章
文章归档
回到顶部