EADST

Check the Index and Token from Tiktoken

Check the Index and Token from Tiktoken

import base64
path = "/home/your_dict_path.tiktoken"
f = open(path, "rb").read()
index = 0
for line in f.splitlines():
    l = line.split()
    print("index: ", l[1])
    print("encode: ", l[0])
    print("decode: ", base64.b64decode(l[0]))
    index += 1
    if index > 20:
        break

Reference Code

相关标签
About Me
XD
Goals determine what you are going to be.
Category
标签云
Markdown Transformers TTS Breakpoint Linux Card 长期记忆 Cloudreve Google v2ray Statistics DeepStream Vim Tracking WAN git Conda scipy FP64 Michelin LoRA Numpy CV Anaconda 飞书 Disk FP8 Jetson Excel Magnet Bipartite Quantization 顶会 Land uWSGI 腾讯云 NameSilo Sklearn IndexTTS2 TSV VSCode 搞笑 关于博主 NLTK printf 净利润 多线程 Base64 图标 FP16 YOLO HuggingFace Pickle CC Clash Firewall Bitcoin Paddle API网关 OpenCV Data 公式 SAM XML hf JSON PyCharm GPTQ icon Knowledge Web BTC 证件照 QWEN llama.cpp FastAPI GPT4 Permission Video Safetensors tqdm 签证 InvalidArgumentError Logo 递归学习法 LLAMA Review RGB LLM ResNet-50 Math Website 第一性原理 Paper Nginx Bert Llama Agent ModelScope VPN MD5 Plotly 音频 SPIE SQL TensorFlow PIP Proxy Jev LaTeX Harness Docker Random CLAP ChatGPT DeepSeek Miniforge Hungarian 强化学习 Mixtral torchinfo SQLite Hilton Rebuttal AI Attention EXCEL 算法题 Claude GGML Pytorch Tensor transformers 个人博客 API Plate CTC OpenAI Translation Streamlit Qwen2.5 PyTorch 论文速读 Qwen2 Diagram 域名 Datetime UNIX Ptyhon Zip CUDA Jupyter Food HaggingFace GoogLeNet COCO git-lfs Use tar PDB XGBoost 财报 mmap Input Password Pillow Search C++ Baidu CAM 多进程 Interview NLP Template Qwen Pandas Heatmap GIT WebCrawler 报税 Quantize RAR 云服务器 Django ONNX FP32 RL CEIR Distillation Hotel 论文 Domain ms-swift Bin Algorithm v0.dev SVR Git Shortcut Ubuntu 继承 LeetCode Dataset 图形思考法 版权 Windows 自我改进 VGG-16 Freesound Image2Text Gemma Animate Augmentation BeautifulSoup TensorRT OCR Color PDF BF16 News Python 阿里云 diffusers Github Tiktoken uwsgi UI CSV FlashAttention logger Vmess Crawler
站点统计

本站现有博文340篇,共被浏览1019313次

本站已经建立2719天!

热门文章
文章归档
回到顶部