NanoGPT Tutorial

环境准备

本文所有操作均在MacBook Air（2020,M1芯片）上测试验证。

安装Miniconda 和Python

在MacOS下，可以通过以下脚本安装：

$ mkdir -p ~/miniconda3
$ curl https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-arm64.sh -o ~/miniconda3/miniconda.sh
$ bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3
$ rm -rf ~/miniconda3/miniconda.sh

安装完成后，可以使用conda命令来管理机器学习的Python环境了。默认系统会自动创建一个Python3.11的环境：

$ python --version
Python 3.11.5
$ whereis python
python: /Users/riguz/miniconda3/bin/python

下载

(base) ➜  nanoGPT git:(master) python data/shakespeare_char/prepare.py
length of dataset in characters: 1,115,394
all the unique characters:
 !$&',-.3:;?ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz
vocab size: 65
train has 1,003,854 tokens
val has 111,540 tokens

python train.py config/train_shakespeare_char.py

# Run it without GPU (mac air), pytorch nightly
# raise AssertionError("Torch not compiled with CUDA enabled")

python train.py config/train_shakespeare_char.py --device=cpu --compile=False --eval_iters=20 --log_interval=1 --block_size=64 --batch_size=12 --n_layer=4 --n_head=4 --n_embd=128 --max_iters=2000 --lr_decay_iters=2000 --dropout=0.0

step 2000: train loss 1.7640, val loss 1.8925
saving checkpoint to out-shakespeare-char
iter 2000: loss 1.6982, time 306.45ms, mfu 0.05%

# total cost： 48s

python sample.py --out_dir=out-shakespeare-char --device=cpu
Overriding: out_dir = out-shakespeare-char
Overriding: device = cpu
number of parameters: 0.80M
Loading meta from data/shakespeare_char/meta.pkl...

I by doth what letterd fain flowarrman,
Lotheefuly daught shouss blate thou his though'd that opt--
Hammine than you, not neme your down way.

ELANUS:
I would and murser wormen that more?
...

python train.py config/train_shakespeare_char.py --device=mps --compile=False --eval_iters=20 --log_interval=1 --block_size=64 --batch_size=12 --n_layer=4 --n_head=4 --n_embd=128 --max_iters=2000 --lr_decay_iters=2000 --dropout=0.0

...
iter 1998: loss 1.8794, time 22.56ms, mfu 0.06%
iter 1999: loss 1.9167, time 22.62ms, mfu 0.06%
step 2000: train loss 1.7640, val loss 1.8925
saving checkpoint to out-shakespeare-char
iter 2000: loss 1.6982, time 352.67ms, mfu 0.05%
# total cost: 51s