NPU/zh
From FriendlyELEC WiKi
Contents
[hide]1 Earlier version RKNPU2 SDK
Link to → v1.5.2
2 Check NPU driver version
$ sudo cat /sys/kernel/debug/rknpu/version RKNPU driver: v0.9.8
3 运行RKNN示例程序
3.1 OS
Tested on the following OS:
3.1.1 Debian11 (bullseye)
- rk3588-sd-debian-bullseye-desktop-6.1-arm64-20250117.img.gz
3.1.2 Ubuntu20 (focal)
- rk3588-sd-ubuntu-focal-desktop-6.1-arm64-20250117.img.gz
3.2 Installing RKNN Runtime
cd ~ export GIT_SSL_NO_VERIFY=1 git clone https://github.com/airockchip/rknn-toolkit2.git cd rknn-toolkit2/rknpu2 sudo cp ./runtime/Linux/librknn_api/aarch64/* /usr/lib sudo cp ./runtime/Linux/rknn_server/aarch64/usr/bin/* /usr/bin/ sudo cp ./runtime/Linux/librknn_api/include/* /usr/include/
3.3 Check rknn version
$ strings /usr/bin/rknn_server |grep 'build@' 2.3.0 (e80ac5c build@2024-11-07T12:52:53) rknn_server version: 2.3.0 (e80ac5c build@2024-11-07T12:52:53) $ strings /usr/lib/librknnrt.so |grep 'librknnrt version:' librknnrt version: 2.3.0 (c949ad889d@2024-11-07T11:35:33)
3.4 C++示例程序: 编译并运行YOLOv5示例
sudo apt-get update sudo apt-get install -y gcc g++ make cmake # fix broken link cd ~/rknn-toolkit2/rknpu2/examples/3rdparty/mpp/Linux/aarch64 rm -f librockchip_mpp.so librockchip_mpp.so.1 ln -s librockchip_mpp.so.0 librockchip_mpp.so ln -s librockchip_mpp.so.0 librockchip_mpp.so.1 cd ~/rknn-toolkit2/rknpu2/examples/rknn_yolov5_demo chmod +x ./build-linux.sh sudo ln -s /usr/bin/gcc /usr/bin/aarch64-gcc sudo ln -s /usr/bin/g++ /usr/bin/aarch64-g++ export GCC_COMPILER=aarch64 ./build-linux.sh -t rk3588 -a aarch64 -b Release cd install/rknn_yolov5_demo_Linux ./rknn_yolov5_demo model/RK3588/yolov5s-640-640.rknn model/bus.jpg
Transfer the generated out.jpg to PC to view the result:
scp out.jpg xxx@YourIP:/tmp/
3.5 Python示例: Debian11下安装并测试RKNN Toolkit Lite2
3.5.1 Install RKNN Toolkit Lite2
sudo apt-get update sudo apt-get install -y python3-dev python3-numpy python3-opencv python3-pip cd ~/rknn-toolkit2 pip3 install ./rknn-toolkit-lite2/packages/rknn_toolkit_lite2-2.3.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl -i https://pypi.tuna.tsinghua.edu.cn/simple/
3.5.2 Run the python example
$ cd ~/rknn-toolkit2/rknn-toolkit-lite2/examples/resnet18/ $ python3 test.py W rknn-toolkit-lite2 version: 2.3.0 --> Load RKNN model done --> Init runtime environment I RKNN: [06:29:43.602] RKNN Runtime Information, librknnrt version: 2.3.0 (c949ad889d@2024-11-07T11:35:33) I RKNN: [06:29:43.602] RKNN Driver Information, version: 0.9.8 I RKNN: [06:29:43.602] RKNN Model Information, version: 6, toolkit version: 2.3.0(compiler version: 2.3.0 (c949ad889d@2024-11-07T11:39:30)), target: RKNPU v2, target platform: rk3588, framework name: PyTorch, framework layout: NCHW, model inference type: static_shape W RKNN: [06:29:43.617] query RKNN_QUERY_INPUT_DYNAMIC_RANGE error, rknn model is static shape type, please export rknn with dynamic_shapes W Query dynamic range failed. Ret code: RKNN_ERR_MODEL_INVALID. (If it is a static shape RKNN model, please ignore the above warning message.) done --> Running model resnet18 -----TOP 5----- [812] score:0.999680 class:"space shuttle" [404] score:0.000249 class:"airliner" [657] score:0.000013 class:"missile" [466] score:0.000009 class:"bullet train, bullet" [895] score:0.000008 class:"warplane, military plane" done
3.6 Python示例: Ubuntu 20.04 (Focal)下安装并测试RKNN Toolkit Lite2
3.6.1 通过编译源代码安装Python 3.9
sudo apt install build-essential libssl-dev libffi-dev software-properties-common \
libbz2-dev libncurses-dev libncursesw5-dev libgdbm-dev liblzma-dev libsqlite3-dev \
tk-dev libgdbm-compat-dev libreadline-dev
wget https://www.python.org/ftp/python/3.9.21/Python-3.9.21.tar.xz
tar -xvf Python-3.9.21.tar.xz
cd Python-3.9.21/
./configure --enable-optimizations
make -j$(nproc)
sudo make install3.6.2 Install RKNN Toolkit Lite2
python3.9 -m pip install --upgrade pip
python3.9 -m pip install opencv-python
cd ~
[ -d rknn-toolkit2 ] || git clone https://github.com/airockchip/rknn-toolkit2.git \
--depth 1 -b master
cd rknn-toolkit2
python3.9 -m pip install \
./rknn-toolkit-lite2/packages/rknn_toolkit_lite2-2.3.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl \
-i https://pypi.tuna.tsinghua.edu.cn/simple/3.6.3 Run the python example
cd ~/rknn-toolkit2/rknn-toolkit-lite2/examples/resnet18/ python3.9 test.py
output:
W rknn-toolkit-lite2 version: 2.3.0 --> Load RKNN model done --> Init runtime environment I RKNN: [07:08:29.756] RKNN Runtime Information, librknnrt version: 2.3.0 (c949ad889d@2024-11-07T11:35:33) I RKNN: [07:08:29.757] RKNN Driver Information, version: 0.9.8 I RKNN: [07:08:29.762] RKNN Model Information, version: 6, toolkit version: 2.3.0(compiler version: 2.3.0 (c949ad889d@2024-11-07T11:39:30)), target: RKNPU v2, target platform: rk3588, framework name: PyTorch, framework layout: NCHW, model inference type: static_shape W RKNN: [07:08:29.814] query RKNN_QUERY_INPUT_DYNAMIC_RANGE error, rknn model is static shape type, please export rknn with dynamic_shapes W Query dynamic range failed. Ret code: RKNN_ERR_MODEL_INVALID. (If it is a static shape RKNN model, please ignore the above warning message.) done --> Running model resnet18 -----TOP 5----- [812] score:0.999680 class:"space shuttle" [404] score:0.000249 class:"airliner" [657] score:0.000013 class:"missile" [466] score:0.000009 class:"bullet train, bullet" [895] score:0.000008 class:"warplane, military plane" done
4 RKLLM的使用与大语言模型
4.1 转换模型 (PC端)
需要将Hugging Face格式的大语言模型转换为RKLLM 模型，使其能够在Rockchip NPU平台上加载使用, 转换操作需要在PC端运行，建议使用的操作系统为Ubuntu20.04，可以使用Docker容器。
4.1.1 安装RKLLM-Toolkit
sudo apt-get update sudo apt-get install -y python3-dev python3-numpy python3-opencv python3-pip cd ~ git clone https://github.com/airockchip/rknn-llm pip3 install rknn-llm/rkllm-toolkit/packages/rkllm_toolkit-1.1.4-cp38-cp38-linux_x86_64.whl
4.1.2 下载并转换模型
使用git clone命令从Hugging Face站点下载所需模型，并修改test.py指向该模型，最后调用test.py开始模型转换:
cd ~/rknn-llm/rkllm-toolkit/examples/ git lfs install git clone https://huggingface.co/THUDM/chatglm3-6b sed -i 's|^modelpath\s*=.*|modelpath = "./chatglm3-6b"|' test.py sed -i 's|^dataset\s*=.*|dataset = None|' test.py python3 test.py
模型转换的输出信息如下:
ubuntu@20de568f4011:~/rknn-llm/rkllm-toolkit/examples$ python3 test.py INFO: rkllm-toolkit version: 1.1.4 The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. Optimizing model: 100%|██████████████████████████████████████████████████████████████████████████████████| 24/24 [00:21<00:00, 1.13it/s] Building model: 100%|█████████████████████████████████████████████████████████████████████████████████| 343/343 [00:01<00:00, 296.74it/s] INFO: The token_id of eos is set to 151643 INFO: The token_id of pad is set to 151643 INFO: The token_id of bos is set to 151643 Converting model: 100%|███████████████████████████████████████████████████████████████████████████| 291/291 [00:00<00:00, 2591385.27it/s] INFO: Exporting the model, please wait .... [=================================================>] 513/513 (100%) INFO: Model has been saved to ./qwen.rkllm!
成功的话会生成文件qwen.rkllm，我们将其改名为chatglm3-6b.rkllm并上传到开发板:
mv qwen.rkllm chatglm3-6b.rkllm scp chatglm3-6b.rkllm root@NanoPC-T6:/home/pi
4.2 编译运行大模型推理示例 (开发板端)
sudo apt-get update sudo apt-get install -y gcc g++ make cmake # get rknn-llm cd ~ [ -d rknn-llm ] || git clone https://github.com/airockchip/rknn-llm cd rknn-llm/examples/rkllm_api_demo chmod +x ./build-linux.sh sed -i 's/^GCC_COMPILER_PATH=.*/GCC_COMPILER_PATH=aarch64-linux-gnu/' ./build-linux.sh ./build-linux.sh # run llm_demo cd ./build/build_linux_aarch64_Release/ export LD_LIBRARY_PATH=~/rknn-llm/rkllm-runtime/Linux/librkllm_api/aarch64:$LD_LIBRARY_PATH ulimit -HSn 10240 # max_new_tokens = 256, max_context_len = 512 taskset f0 ./llm_demo ~/chatglm3-6b.rkllm 256 512
运行结果:
rkllm init start I rkllm: rkllm-runtime version: 1.1.4, rknpu driver version: 0.9.8, platform: RK3588 rkllm init success **********************可输入以下问题对应序号获取回答/或自定义输入******************** [0] 把下面的现代文翻译成文言文: 到了春风和煦，阳光明媚的时候，湖面平静，没有惊涛骇浪，天色湖光相连，一片碧绿，广阔无际；沙洲上的鸥鸟，时而飞翔，时而停歇，美丽的鱼游来游去，岸上与小洲上的花草，青翠欲滴。 [1] 以咏梅为题目，帮我写一首古诗，要求包含梅花、白雪等元素。 [2] 上联: 江边惯看千帆过 [3] 把这句话翻译成中文: Knowledge can be acquired from many sources. These include books, teachers and practical experience, and each has its own advantages. The knowledge we gain from books and formal education enables us to learn about things that we have no opportunity to experience in daily life. We can also develop our analytical skills and learn how to view and interpret the world around us in different ways. Furthermore, we can learn from the past by reading books. In this way, we won't repeat the mistakes of others and can build on their achievements. [4] 把这句话翻译成英文: RK3588是新一代高端处理器，具有高算力、低功耗、超强多媒体、丰富数据接口等特点 ************************************************************************* user: 天空为什么是蓝色的? robot: 蓝色是如何形成的? 天空为什么是蓝色的？这是因为大气层中的气体分子对太阳光的吸收和散射。当太阳光穿过大气层时，其中的可见光波（包括红、橙、黄、绿、蓝、靛、紫等颜色）与气体分子相互作用。其中，蓝色光的波长较短，能量较高。当蓝色光与气体分子相互作用时，它们被散射到各个方向，使得天空呈现出蓝色。 此外，太阳光中的蓝色光相对于其他颜色的光更容易被散射。这是因为在太阳光谱中，蓝色光的波长较短，频率较高。当光线穿过大气层时，空气中的气体分子和悬浮颗粒会对不同波长的光进行散射，而蓝色光的波长较短，能量较高，因此更容易被散射。这使得天空呈现出蓝色。 user:
4.3 有用的资源
5 Doc
https://github.com/rockchip-linux/rknpu2/tree/master/doc
6 Other
6.1 查看NPU占有率
sudo cat /sys/kernel/debug/rknpu/load
6.2 Set NPU freq
echo userspace | sudo tee /sys/class/devfreq/fdab0000.npu/governor > /dev/null echo 800000000 | sudo tee /sys/class/devfreq/fdab0000.npu/min_freq > /dev/null echo 1000000000 | sudo tee /sys/class/devfreq/fdab0000.npu/max_freq > /dev/null
6.3 查看NPU频率
cat /sys/class/devfreq/fdab0000.npu/cur_freq
