EADST

QWEN7B to LLAMA7B Model Structure

Here is the markdown format for the LLAMA7B model structure, detailing each layer and component:


LLAMA7B Model Structure

The LLAMA7B model consists of the following layers and components:

Embedding Layer

  • model.embed_tokens.weight: torch.Size([151851, 4096])

Layers

Each layer in the model has the following components:

Layer 0 to Layer 31

Each layer (model.layers.[0-31]) includes:

  • input_layernorm.weight: torch.Size([4096])

  • Self-Attention Sublayer:

    • q_proj.weight: torch.Size([4096, 4096])

    • k_proj.weight: torch.Size([4096, 4096])

    • v_proj.weight: torch.Size([4096, 4096])

    • q_proj.bias: torch.Size([4096])

    • k_proj.bias: torch.Size([4096])

    • v_proj.bias: torch.Size([4096])

    • o_proj.weight: torch.Size([4096, 4096])

    • post_attention_layernorm.weight: torch.Size([4096])

  • MLP (Multi-Layer Perceptron) Sublayer:

    • up_proj.weight: torch.Size([11008, 4096])

    • gate_proj.weight: torch.Size([11008, 4096])

    • down_proj.weight: torch.Size([4096, 11008])

Final Layer Normalization and Output

  • model.norm.weight: torch.Size([4096])
  • lm_head.weight: torch.Size([151851, 4096])
相关标签
About Me
XD
Goals determine what you are going to be.
Category
标签云
WebCrawler Distillation XGBoost PyTorch Proxy CEIR TTS JSON hf 域名 Disk 报税 Ubuntu Llama SQL Search Magnet Bipartite SPIE PIP 论文 CTC CUDA Markdown git-lfs BF16 FP8 ONNX IndexTTS2 Bin Image2Text 搞笑 Zip 算法题 Qwen2.5 VPN Pytorch tqdm 音频 API Quantization Nginx News 个人博客 XML Quantize QWEN Mixtral Tiktoken Translation OpenAI Use Qwen2 Card uWSGI 腾讯云 Git Permission YOLO Bitcoin Random 财报 FP32 Plate Review GPT4 Knowledge mmap LLM LLAMA Jupyter 净利润 Bert Hungarian LaTeX 强化学习 Cloudreve tar NameSilo CLAP Datetime Safetensors UNIX Math logger Anaconda RGB Input Firewall Harness Logo TSV 多进程 NLP Qwen Hilton EXCEL Tensor LoRA 长期记忆 PyCharm Paper 云服务器 GoogLeNet TensorRT 版权 FastAPI 多线程 CSV Website icon 证件照 VGG-16 Pandas Statistics C++ Excel 关于博主 Linux RL v2ray ResNet-50 printf 论文速读 Land Crawler llama.cpp TensorFlow 阿里云 Github UI Jev 递归学习法 GPTQ 第一性原理 Paddle uwsgi torchinfo WAN ms-swift Password Template Django GGML Data MD5 Pillow OCR 顶会 Diagram 飞书 Shortcut git Numpy Agent InvalidArgumentError Freesound Transformers Augmentation Miniforge GIT Tracking Streamlit Pickle Vmess HuggingFace Ptyhon CC Gemma 自我改进 RAR Animate Video 公式 BeautifulSoup Interview FP16 DeepStream PDF Color Algorithm Web Domain CV Claude AI API网关 Breakpoint BTC 图形思考法 Jetson v0.dev Clash 继承 SQLite VSCode Conda LeetCode Google transformers 签证 Attention HaggingFace 图标 ModelScope Heatmap SAM Michelin Food Windows SVR CAM scipy FP64 Dataset diffusers Plotly Rebuttal NLTK Python FlashAttention OpenCV Base64 Hotel DeepSeek Docker COCO Vim PDB ChatGPT Sklearn Baidu
站点统计

本站现有博文340篇,共被浏览1018372次

本站已经建立2718天!

热门文章
文章归档
回到顶部