EADST

QWEN7B to LLAMA GPTQ model structure

Here is the markdown format for the GPTQ model structure, detailing each layer and component:


GPTQ Model Structure

The GPTQ model consists of the following layers and components:

Embedding Layer

  • model.embed_tokens.weight: torch.Size([151851, 4096])

Layers

Each layer in the model has the following components:

Layer 0 to Layer 31

Each layer (model.layers.[0-31]) includes:

  • input_layernorm.weight: torch.Size([4096])

  • Self-Attention Sublayer:

    • k_proj:

      • qweight: torch.Size([512, 4096])

      • qzeros: torch.Size([32, 512])

      • scales: torch.Size([32, 4096])

      • g_idx: torch.Size([4096])

      • bias: torch.Size([4096])

    • o_proj:

      • qweight: torch.Size([512, 4096])

      • qzeros: torch.Size([32, 512])

      • scales: torch.Size([32, 4096])

      • g_idx: torch.Size([4096])

      • bias: torch.Size([4096])

    • q_proj:

      • qweight: torch.Size([512, 4096])

      • qzeros: torch.Size([32, 512])

      • scales: torch.Size([32, 4096])

      • g_idx: torch.Size([4096])

      • bias: torch.Size([4096])

    • v_proj:

      • qweight: torch.Size([512, 4096])

      • qzeros: torch.Size([32, 512])

      • scales: torch.Size([32, 4096])

      • g_idx: torch.Size([4096])

      • bias: torch.Size([4096])

  • MLP (Multi-Layer Perceptron) Sublayer:

    • down_proj:

      • qweight: torch.Size([1376, 4096])

      • qzeros: torch.Size([86, 512])

      • scales: torch.Size([86, 4096])

      • g_idx: torch.Size([11008])

      • bias: torch.Size([4096])

    • gate_proj:

      • qweight: torch.Size([512, 11008])

      • qzeros: torch.Size([32, 1376])

      • scales: torch.Size([32, 11008])

      • g_idx: torch.Size([4096])

      • bias: torch.Size([11008])

    • up_proj:

      • qweight: torch.Size([512, 11008])

      • qzeros: torch.Size([32, 1376])

      • scales: torch.Size([32, 11008])

      • g_idx: torch.Size([4096])

      • bias: torch.Size([11008])

  • post_attention_layernorm.weight: torch.Size([4096])

Final Layer Normalization and Output

  • model.norm.weight: torch.Size([4096])
  • lm_head.weight: torch.Size([151851, 4096])
相关标签
About Me
XD
Goals determine what you are going to be.
Category
标签云
Zip Magnet Safetensors Animate TensorRT ChatGPT Plate Statistics ResNet-50 IndexTTS2 递归学习法 SPIE Tiktoken Shortcut hf PIP Windows Data Docker Vim ONNX PDB CAM torchinfo 阿里云 Card 多进程 Bitcoin Paper Domain TSV 公式 TTS Gemma News Qwen2.5 v2ray 版权 Use LoRA 签证 Clash diffusers ModelScope Logo Transformers BTC 长期记忆 PyCharm LLM scipy CEIR Math Augmentation GIT WebCrawler Disk XML 继承 FP16 InvalidArgumentError Pytorch 论文 LeetCode mmap FP8 UI 第一性原理 MD5 Website 图标 Jupyter printf VSCode Permission Git Proxy 自我改进 Llama FastAPI VPN Vmess Distillation 个人博客 报税 Agent XGBoost RGB AI FP32 Conda Miniforge DeepStream Review GGML Excel Tracking GoogLeNet Firewall Knowledge uWSGI Diagram Tensor Cloudreve 财报 NameSilo Color Interview 论文速读 Algorithm WAN Bert Streamlit 多线程 Sklearn Input Heatmap NLP Bipartite icon CV ms-swift Pickle HuggingFace Google Image2Text Jev Hilton OpenCV LLAMA FP64 NLTK BF16 Quantization SVR RAR FlashAttention HaggingFace PDF Breakpoint 搞笑 SAM 飞书 OpenAI YOLO 证件照 域名 llama.cpp C++ Crawler 图形思考法 transformers Pillow Qwen2 Dataset Claude Anaconda SQLite Ubuntu VGG-16 Baidu Harness API Github Nginx Pandas DeepSeek Datetime TensorFlow Markdown 顶会 Django UNIX SQL 算法题 强化学习 Quantize uwsgi Random tar Rebuttal QWEN Video Web Michelin Food v0.dev CLAP JSON Hotel Mixtral git-lfs logger Search BeautifulSoup Ptyhon Hungarian Paddle RL Numpy 音频 Python tqdm 云服务器 Base64 Freesound EXCEL GPT4 Template CC 腾讯云 Attention COCO OCR Land Jetson git GPTQ Bin PyTorch Password CTC CUDA CSV LaTeX Qwen 净利润 Translation 关于博主 API网关 Plotly Linux
站点统计

本站现有博文340篇,共被浏览1017851次

本站已经建立2718天!

热门文章
文章归档
回到顶部