EADST

Attention Net with Pytorch

Attention net may be put after the LSTM processing in the NLP task.

import torch
import torch.nn as nn
from torch.autograd import Variable

# attention layer
def attention_net(lstm_output):
    hidden_size = 300
    w_omega = Variable(torch.zeros(hidden_size, 2))
    u_omega = Variable(torch.zeros(2))
    output_reshape = torch.Tensor.reshape(lstm_output, [-1, hidden_size])
    u = torch.tanh(torch.mm(output_reshape, w_omega))
    attn_hidden_layer = torch.mm(u, torch.Tensor.reshape(u_omega, [-1, 1]))
    sequence_length = lstm_output.size()[1]
    alphas = nn.functional.softmax(attn_hidden_layer, dim=1)
    alphas_reshape = torch.Tensor.reshape(alphas, [-1, sequence_length, 1])
    state = lstm_output.permute(1, 0, 2)
    attn_output = torch.sum(state * alphas_reshape, 1)
    return attn_output


# add attention layer after lstm
if attetion_mode == True:
    lstm_out = lstm_out.permute(1, 0, 2)
    lstm_out = attention_net(lstm_out)

相关标签
About Me
XD
Goals determine what you are going to be.
Category
标签云
Math BF16 自我改进 logger Transformers Google icon VGG-16 音频 Excel Proxy CSV Firewall Agent TensorFlow hf Bipartite 阿里云 Jetson SPIE Safetensors tar Claude 公式 Tensor BTC Template FlashAttention FP64 强化学习 GGML Qwen2 Color PIP 算法题 净利润 mmap Cloudreve Pytorch Datetime AI Disk Llama Use v0.dev 图标 Clash OCR LaTeX Bert v2ray Windows MD5 Streamlit RL CTC 签证 Shortcut ModelScope QWEN PDF 腾讯云 Mixtral Paddle TensorRT LoRA Numpy Land 继承 Algorithm Vim torchinfo Ptyhon Search 长期记忆 搞笑 Django Knowledge NLP Video Python Food HaggingFace Pandas Hotel Translation DeepSeek Image2Text WAN SQL CLAP JSON COCO XGBoost Rebuttal scipy LeetCode GPT4 CV Quantization Review SQLite NameSilo 财报 域名 tqdm SAM Qwen PyCharm Jev LLM 多进程 IndexTTS2 图形思考法 RGB Interview Harness printf Dataset Base64 Quantize Web FastAPI Magnet GIT Animate Input GPTQ NLTK Card Pillow 第一性原理 OpenCV News Bitcoin Data Sklearn 报税 Markdown Linux CUDA Nginx Zip 云服务器 Jupyter WebCrawler Conda llama.cpp EXCEL VSCode Attention TSV ResNet-50 DeepStream Plotly Tiktoken CC 论文 GoogLeNet SVR Logo 飞书 PyTorch HuggingFace Ubuntu FP16 Crawler Qwen2.5 关于博主 Bin git C++ Github uWSGI Breakpoint Freesound CAM Distillation Tracking XML transformers Website Baidu LLAMA Git 递归学习法 API网关 Hungarian Plate CEIR FP32 个人博客 API 论文速读 Gemma Permission ChatGPT Random Password Michelin 多线程 Diagram OpenAI UI ONNX Heatmap RAR ms-swift Hilton Augmentation Docker 顶会 VPN 证件照 InvalidArgumentError UNIX PDB YOLO Pickle BeautifulSoup Domain FP8 版权 uwsgi git-lfs Miniforge Statistics Vmess TTS Paper diffusers Anaconda
站点统计

本站现有博文340篇,共被浏览1017816次

本站已经建立2718天!

热门文章
文章归档
回到顶部