EADST

GGML Q4_0 Quantize Analysis in llama.cpp

GGML Q4_0 Quantization in llama.cpp

For the LLAMA7B model, there are 387 tensors consisting of various weights and biases. These tensors include token_embd.weight, 32 sets of attention and feedforward network weights and biases (attn_norm.weigh, attn_q.weight, attn_k.weight, attn_v.weight, attn_q.bias, attn_k.bias, attn_v.bias, attn_output.weight, ffn_norm.weight, ffn_up.weight, ffn_gate.weight, ffn_down.weight), output_norm.weight, and output.weight.

Quantization Details:

  • Total Tensors for Quantization: 226
  • token_embd.weight
  • 32 sets of: attn_q.weight, attn_k.weight, attn_v.weight, attn_output.weight, ffn_up.weight, ffn_gate.weight, ffn_down.weight
  • output.weight

Tensor Breakdown:

  • llama_model_loader:
  • f32 type: 161 tensors
  • f16 type: 226 tensors
  • llama_model_quantize_internal:
  • Meta size: 6162784 bytes

Example Tensors:

  • [ 1/ 387] token_embd.weight - [ 4096, 151851, 1, 1], type = f16, quantizing to q4_0 .. size = 1186.34 MB -> 333.66 MB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021

[ 2/ 387] blk.0.attn_norm.weight - [ 4096, 1, 1, 1], type = f32, size = 0.016 MB

  • [ 3/ 387] blk.0.attn_q.weight - [ 4096, 4096, 1, 1], type = f16, quantizing to q4_0 .. size = 32.00 MB -> 9.00 MB | hist: 0.036 0.015 0.025 0.038 0.056 0.076 0.097 0.113 0.120 0.113 0.097 0.076 0.056 0.038 0.025 0.020

  • [ 4/ 387] blk.0.attn_k.weight - [ 4096, 4096, 1, 1], type = f16, quantizing to q4_0 .. size = 32.00 MB -> 9.00 MB | hist: 0.036 0.015 0.024 0.037 0.055 0.075 0.097 0.115 0.123 0.115 0.097 0.076 0.055 0.037 0.024 0.020

  • [ 5/ 387] blk.0.attn_v.weight - [ 4096, 4096, 1, 1], type = f16, quantizing to q4_0 .. size = 32.00 MB -> 9.00 MB | hist: 0.036 0.016 0.025 0.039 0.056 0.076 0.096 0.112 0.119 0.112 0.096 0.076 0.056 0.039 0.025 0.021

[ 6/ 387] blk.0.attn_q.bias - [ 4096, 1, 1, 1], type = f32, size = 0.016 MB

[ 7/ 387] blk.0.attn_k.bias - [ 4096, 1, 1, 1], type = f32, size = 0.016 MB

[ 8/ 387] blk.0.attn_v.bias - [ 4096, 1, 1, 1], type = f32, size = 0.016 MB

  • [ 9/ 387] blk.0.attn_output.weight - [ 4096, 4096, 1, 1], type = f16, quantizing to q4_0 .. size = 32.00 MB -> 9.00 MB | hist: 0.036 0.015 0.025 0.039 0.056 0.077 0.096 0.112 0.118 0.112 0.096 0.077 0.056 0.039 0.025 0.021

[ 10/ 387] blk.0.ffn_norm.weight - [ 4096, 1, 1, 1], type = f32, size = 0.016 MB

  • [ 11/ 387] blk.0.ffn_up.weight - [ 4096, 11008, 1, 1], type = f16, quantizing to q4_0 .. size = 86.00 MB -> 24.19 MB | hist: 0.037 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.116 0.111 0.096 0.077 0.057 0.039 0.025 0.021

  • [ 12/ 387] blk.0.ffn_gate.weight - [ 4096, 11008, 1, 1], type = f16, quantizing to q4_0 .. size = 86.00 MB -> 24.19 MB | hist: 0.037 0.016 0.026 0.039 0.057 0.077 0.096 0.110 0.116 0.110 0.096 0.077 0.057 0.040 0.026 0.021

  • [ 13/ 387] blk.0.ffn_down.weight - [11008, 4096, 1, 1], type = f16, quantizing to q4_0 .. size = 86.00 MB -> 24.19 MB | hist: 0.036 0.016 0.025 0.039 0.057 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021

*...and so on for other tensors [14/ 387]-[385/ 387] *

The remaining 31 blocks follow a similar pattern. blk.0*-blk.31*

[ 386/ 387] output_norm.weight - [ 4096, 1, 1, 1], type = f32, size = 0.016 MB

  • [ 387/ 387] output.weight - [ 4096, 151851, 1, 1], type = f16, quantizing to q6_K .. size = 1186.34 MB -> 486.58 MB | hist:

llama_model_quantize_internal: model size = 14727.19 MB

llama_model_quantize_internal: quant size = 4296.76 MB

llama_model_quantize_internal: hist: 0.036 0.016 0.025 0.039 0.056 0.077 0.096 0.111 0.117 0.111 0.096 0.077 0.057 0.039 0.025 0.021

Reference:

相关标签
About Me
XD
Goals determine what you are going to be.
Category
标签云
tar Python Firewall Freesound Base64 报税 Cloudreve Breakpoint 个人博客 GPT4 ms-swift Translation CAM Attention QWEN Transformers GoogLeNet Heatmap 净利润 证件照 transformers Password torchinfo 腾讯云 Random Qwen2 Distillation Paddle Docker Vmess Django HaggingFace LoRA ModelScope Linux 论文速读 Card UI CSV Bitcoin OpenCV API FastAPI Harness Markdown SAM News LeetCode Nginx Math Sklearn Qwen LaTeX Mixtral RGB Safetensors Video Plotly JSON FP16 ONNX Review SQL OpenAI RL Clash FP64 Gemma Quantization Conda 签证 Image2Text FP8 YOLO Michelin 版权 Domain NLP Bipartite Data LLAMA 关于博主 自我改进 Disk RAR InvalidArgumentError Website Tracking GIT Pickle v0.dev XML Windows Animate PIP scipy 图形思考法 Jetson Quantize 强化学习 Github WAN Shortcut Input Pandas SPIE GPTQ BTC diffusers FP32 顶会 音频 CLAP Ptyhon 多线程 ChatGPT CEIR CTC Anaconda 算法题 Hotel NLTK Algorithm Magnet Datetime Pytorch Excel Miniforge IndexTTS2 Hungarian TTS Logo 飞书 API网关 域名 v2ray Interview DeepSeek Land SVR 长期记忆 printf Food 第一性原理 继承 图标 TensorFlow hf DeepStream tqdm git-lfs Google TensorRT Bin Color Web Statistics VGG-16 Ubuntu BF16 Dataset UNIX 财报 Llama uwsgi Bert Paper FlashAttention 论文 COCO mmap Claude Proxy 递归学习法 云服务器 WebCrawler Diagram 公式 多进程 Hilton PyCharm Qwen2.5 PDB TSV Baidu LLM ResNet-50 Git VPN Use Template AI uWSGI Vim Numpy PyTorch CUDA Crawler NameSilo git Plate XGBoost Agent CC PDF llama.cpp Augmentation Tensor Pillow SQLite EXCEL Jev CV BeautifulSoup Zip Rebuttal icon OCR GGML HuggingFace 搞笑 Tiktoken Knowledge Permission Search VSCode Jupyter 阿里云 C++ MD5 logger Streamlit
站点统计

本站现有博文340篇,共被浏览1019339次

本站已经建立2719天!

热门文章
文章归档
回到顶部