NPU
Contents
[hide]1 Earlier version RKNPU2 SDK
Link to → v1.5.2
2 How to test RKNN
2.1 OS
Tested on the following OS:
2.1.1 Debian11 (bullseye)
- rk3588-sd-debian-bullseye-desktop-6.1-arm64-20250117.img.gz
2.1.2 Ubuntu20 (focal)
- rk3588-sd-ubuntu-focal-desktop-6.1-arm64-20250117.img.gz
2.2 Check NPU driver version
$ sudo cat /sys/kernel/debug/rknpu/version RKNPU driver: v0.9.8
2.3 install rknpu
cd ~ export GIT_SSL_NO_VERIFY=1 git clone https://github.com/airockchip/rknn-toolkit2.git cd rknn-toolkit2/rknpu2 sudo cp ./runtime/Linux/librknn_api/aarch64/* /usr/lib sudo cp ./runtime/Linux/rknn_server/aarch64/usr/bin/* /usr/bin/ sudo cp ./runtime/Linux/librknn_api/include/* /usr/include/
2.4 check rknn version
$ strings /usr/bin/rknn_server |grep 'build@' 2.3.0 (e80ac5c build@2024-11-07T12:52:53) rknn_server version: 2.3.0 (e80ac5c build@2024-11-07T12:52:53) $ strings /usr/lib/librknnrt.so |grep 'librknnrt version:' librknnrt version: 2.3.0 (c949ad889d@2024-11-07T11:35:33)
2.5 run rknn_yolov5_demo
sudo apt-get update sudo apt-get install -y gcc g++ make cmake # fix broken link cd ~/rknn-toolkit2/rknpu2/examples/3rdparty/mpp/Linux/aarch64 rm -f librockchip_mpp.so librockchip_mpp.so.1 ln -s librockchip_mpp.so.0 librockchip_mpp.so ln -s librockchip_mpp.so.0 librockchip_mpp.so.1 cd ~/rknn-toolkit2/rknpu2/examples/rknn_yolov5_demo chmod +x ./build-linux.sh sudo ln -s /usr/bin/gcc /usr/bin/aarch64-gcc sudo ln -s /usr/bin/g++ /usr/bin/aarch64-g++ export GCC_COMPILER=aarch64 ./build-linux.sh -t rk3588 -a aarch64 -b Release cd install/rknn_yolov5_demo_Linux ./rknn_yolov5_demo model/RK3588/yolov5s-640-640.rknn model/bus.jpg
Transfer the generated out.jpg to PC to view the result:
scp out.jpg xxx@YourIP:/tmp/
2.6 install rknn_toolkit on debian11
2.6.1 install rknn_toolkit
sudo apt-get update sudo apt-get install -y python3-dev python3-numpy python3-opencv python3-pip cd ~/rknn-toolkit2 pip3 install ./rknn-toolkit-lite2/packages/rknn_toolkit_lite2-2.3.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl -i https://pypi.tuna.tsinghua.edu.cn/simple/
2.6.2 run python demo
$ cd ~/rknn-toolkit2/rknn-toolkit-lite2/examples/resnet18/ $ python3 test.py W rknn-toolkit-lite2 version: 2.3.0 --> Load RKNN model done --> Init runtime environment I RKNN: [06:29:43.602] RKNN Runtime Information, librknnrt version: 2.3.0 (c949ad889d@2024-11-07T11:35:33) I RKNN: [06:29:43.602] RKNN Driver Information, version: 0.9.8 I RKNN: [06:29:43.602] RKNN Model Information, version: 6, toolkit version: 2.3.0(compiler version: 2.3.0 (c949ad889d@2024-11-07T11:39:30)), target: RKNPU v2, target platform: rk3588, framework name: PyTorch, framework layout: NCHW, model inference type: static_shape W RKNN: [06:29:43.617] query RKNN_QUERY_INPUT_DYNAMIC_RANGE error, rknn model is static shape type, please export rknn with dynamic_shapes W Query dynamic range failed. Ret code: RKNN_ERR_MODEL_INVALID. (If it is a static shape RKNN model, please ignore the above warning message.) done --> Running model resnet18 -----TOP 5----- [812] score:0.999680 class:"space shuttle" [404] score:0.000249 class:"airliner" [657] score:0.000013 class:"missile" [466] score:0.000009 class:"bullet train, bullet" [895] score:0.000008 class:"warplane, military plane" done
2.7 install rknn_toolkit on ubuntu-focal
2.7.1 build python3.9 from source
sudo apt install build-essential libssl-dev libffi-dev software-properties-common \ libbz2-dev libncurses-dev libncursesw5-dev libgdbm-dev liblzma-dev libsqlite3-dev \ tk-dev libgdbm-compat-dev libreadline-dev wget https://www.python.org/ftp/python/3.9.21/Python-3.9.21.tar.xz tar -xvf Python-3.9.21.tar.xz cd Python-3.9.21/ ./configure --enable-optimizations make -j$(nproc) sudo make install
2.7.2 install rknn_toolkit
python3.9 -m pip install --upgrade pip python3.9 -m pip install opencv-python cd ~ [ -d rknn-toolkit2 ] || git clone https://github.com/airockchip/rknn-toolkit2.git \ --depth 1 -b master cd rknn-toolkit2 python3.9 -m pip install \ ./rknn-toolkit-lite2/packages/rknn_toolkit_lite2-2.3.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl \ -i https://pypi.tuna.tsinghua.edu.cn/simple/
2.7.3 run python demo on ubuntu-focal
cd ~/rknn-toolkit2/rknn-toolkit-lite2/examples/resnet18/ python3.9 test.py
output:
W rknn-toolkit-lite2 version: 2.3.0 --> Load RKNN model done --> Init runtime environment I RKNN: [07:08:29.756] RKNN Runtime Information, librknnrt version: 2.3.0 (c949ad889d@2024-11-07T11:35:33) I RKNN: [07:08:29.757] RKNN Driver Information, version: 0.9.8 I RKNN: [07:08:29.762] RKNN Model Information, version: 6, toolkit version: 2.3.0(compiler version: 2.3.0 (c949ad889d@2024-11-07T11:39:30)), target: RKNPU v2, target platform: rk3588, framework name: PyTorch, framework layout: NCHW, model inference type: static_shape W RKNN: [07:08:29.814] query RKNN_QUERY_INPUT_DYNAMIC_RANGE error, rknn model is static shape type, please export rknn with dynamic_shapes W Query dynamic range failed. Ret code: RKNN_ERR_MODEL_INVALID. (If it is a static shape RKNN model, please ignore the above warning message.) done --> Running model resnet18 -----TOP 5----- [812] score:0.999680 class:"space shuttle" [404] score:0.000249 class:"airliner" [657] score:0.000013 class:"missile" [466] score:0.000009 class:"bullet train, bullet" [895] score:0.000008 class:"warplane, military plane" done
3 RKLLM and Large Language Models
3.1 Model Conversion (PC Side)
The Hugging Face-format large language model needs to be converted into an RKLLM model to enable loading and usage on the Rockchip NPU platform. The conversion process must be performed on a PC, with Ubuntu 20.04 as the recommended operating system. A Docker container can also be used.
3.1.1 Installing RKLLM-Toolkit
sudo apt-get update sudo apt-get install -y python3-dev python3-numpy python3-opencv python3-pip cd ~ git clone https://github.com/airockchip/rknn-llm pip3 install rknn-llm/rkllm-toolkit/packages/rkllm_toolkit-1.1.4-cp38-cp38-linux_x86_64.whl
3.1.2 Download and Convert the Model
Use the git clone command to download the required model from the Hugging Face repository. Update test.py to point to the downloaded model, then execute test.py to start the model conversion process:
cd ~/rknn-llm/rkllm-toolkit/examples/ git lfs install git clone https://huggingface.co/THUDM/chatglm3-6b sed -i 's|^modelpath\s*=.*|modelpath = "./chatglm3-6b"|' test.py sed -i 's|^dataset\s*=.*|dataset = None|' test.py python3 test.py
The output information of the model conversion is as follows:
ubuntu@20de568f4011:~/rknn-llm/rkllm-toolkit/examples$ python3 test.py INFO: rkllm-toolkit version: 1.1.4 The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. Optimizing model: 100%|██████████████████████████████████████████████████████████████████████████████████| 24/24 [00:21<00:00, 1.13it/s] Building model: 100%|█████████████████████████████████████████████████████████████████████████████████| 343/343 [00:01<00:00, 296.74it/s] INFO: The token_id of eos is set to 151643 INFO: The token_id of pad is set to 151643 INFO: The token_id of bos is set to 151643 Converting model: 100%|███████████████████████████████████████████████████████████████████████████| 291/291 [00:00<00:00, 2591385.27it/s] INFO: Exporting the model, please wait .... [=================================================>] 513/513 (100%) INFO: Model has been saved to ./qwen.rkllm!
If successful, the file qwen.rkllm will be generated. Rename it to chatglm3-6b.rkllm and upload it to the development board:
mv qwen.rkllm chatglm3-6b.rkllm scp chatglm3-6b.rkllm root@NanoPC-T6:/home/pi
3.2 Compile and run demo application (On Development Board)
sudo apt-get update sudo apt-get install -y gcc g++ make cmake # get rknn-llm cd ~ [ -d rknn-llm ] || git clone https://github.com/airockchip/rknn-llm cd rknn-llm/examples/rkllm_api_demo chmod +x ./build-linux.sh sed -i 's/^GCC_COMPILER_PATH=.*/GCC_COMPILER_PATH=aarch64-linux-gnu/' ./build-linux.sh ./build-linux.sh # run llm_demo cd ./build/build_linux_aarch64_Release/ export LD_LIBRARY_PATH=~/rknn-llm/rkllm-runtime/Linux/librkllm_api/aarch64:$LD_LIBRARY_PATH ulimit -HSn 10240 # max_new_tokens = 256, max_context_len = 512 taskset f0 ./llm_demo ~/chatglm3-6b.rkllm 256 512
Output:
rkllm init start I rkllm: rkllm-runtime version: 1.1.4, rknpu driver version: 0.9.8, platform: RK3588 rkllm init success **********************可输入以下问题对应序号获取回答/或自定义输入******************** [0] 把下面的现代文翻译成文言文: 到了春风和煦,阳光明媚的时候,湖面平静,没有惊涛骇浪,天色湖光相连,一片碧绿,广阔无际;沙洲上的鸥鸟,时而飞翔,时而停歇,美丽的鱼游来游去,岸上与小洲上的花草,青翠欲滴。 [1] 以咏梅为题目,帮我写一首古诗,要求包含梅花、白雪等元素。 [2] 上联: 江边惯看千帆过 [3] 把这句话翻译成中文: Knowledge can be acquired from many sources. These include books, teachers and practical experience, and each has its own advantages. The knowledge we gain from books and formal education enables us to learn about things that we have no opportunity to experience in daily life. We can also develop our analytical skills and learn how to view and interpret the world around us in different ways. Furthermore, we can learn from the past by reading books. In this way, we won't repeat the mistakes of others and can build on their achievements. [4] 把这句话翻译成英文: RK3588是新一代高端处理器,具有高算力、低功耗、超强多媒体、丰富数据接口等特点 ************************************************************************* user: 天空为什么是蓝色的? robot: 蓝色是如何形成的? 天空为什么是蓝色的?这是因为大气层中的气体分子对太阳光的吸收和散射。当太阳光穿过大气层时,其中的可见光波(包括红、橙、黄、绿、蓝、靛、紫等颜色)与气体分子相互作用。其中,蓝色光的波长较短,能量较高。当蓝色光与气体分子相互作用时,它们被散射到各个方向,使得天空呈现出蓝色。 此外,太阳光中的蓝色光相对于其他颜色的光更容易被散射。这是因为在太阳光谱中,蓝色光的波长较短,频率较高。当光线穿过大气层时,空气中的气体分子和悬浮颗粒会对不同波长的光进行散射,而蓝色光的波长较短,能量较高,因此更容易被散射。这使得天空呈现出蓝色。 user:
3.3 Useful Resources
4 Doc
https://github.com/rockchip-linux/rknpu2/tree/master/doc
5 Other
5.1 View NPU Load
sudo cat /sys/kernel/debug/rknpu/load
5.2 View NPU freq
cat /sys/class/devfreq/fdab0000.npu/cur_freq