Difference between revisions of "NPU"

From FriendlyELEC WiKi
Jump to: navigation, search
(updated by API)
(updated by API)
Line 3: Line 3:
 
==Earlier version RKNPU2 SDK==
 
==Earlier version RKNPU2 SDK==
 
Link to → [[NPU_SDK1.5.2|v1.5.2]]
 
Link to → [[NPU_SDK1.5.2|v1.5.2]]
==How to test NPU==
+
==How to test RKNN==
 
===OS===
 
===OS===
 
Tested on the following OS:
 
Tested on the following OS:
 
====Debian11 (bullseye)====
 
====Debian11 (bullseye)====
* rk3588-sd-debian-bullseye-desktop-6.1-arm64-20241011.img.gz
+
* rk3588-sd-debian-bullseye-desktop-6.1-arm64-20250117.img.gz
 
====Ubuntu20 (focal)====
 
====Ubuntu20 (focal)====
* rk3588-sd-ubuntu-focal-desktop-6.1-arm64-20241011.img.gz
+
* rk3588-sd-ubuntu-focal-desktop-6.1-arm64-20250117.img.gz
 +
===Check NPU driver version===
 +
<syntaxhighlight lang="text">
 +
$ sudo cat /sys/kernel/debug/rknpu/version
 +
RKNPU driver: v0.9.8
 +
</syntaxhighlight>
 
===install rknpu===
 
===install rknpu===
 
<syntaxhighlight lang="text">
 
<syntaxhighlight lang="text">
 
cd ~
 
cd ~
 
export GIT_SSL_NO_VERIFY=1
 
export GIT_SSL_NO_VERIFY=1
git clone https://github.com/airockchip/rknn-toolkit2.git --depth 1 -b master
+
git clone https://github.com/airockchip/rknn-toolkit2.git
 
cd rknn-toolkit2/rknpu2
 
cd rknn-toolkit2/rknpu2
 
sudo cp ./runtime/Linux/librknn_api/aarch64/* /usr/lib
 
sudo cp ./runtime/Linux/librknn_api/aarch64/* /usr/lib
Line 20: Line 25:
 
sudo cp ./runtime/Linux/librknn_api/include/* /usr/include/
 
sudo cp ./runtime/Linux/librknn_api/include/* /usr/include/
 
</syntaxhighlight>
 
</syntaxhighlight>
 
 
===check rknn version===
 
===check rknn version===
 
<syntaxhighlight lang="text">
 
<syntaxhighlight lang="text">
 
$ strings /usr/bin/rknn_server |grep 'build@'
 
$ strings /usr/bin/rknn_server |grep 'build@'
2.2.0 (3e10681 build@2024-09-06T10:26:49)
+
2.3.0 (e80ac5c build@2024-11-07T12:52:53)
rknn_server version: 2.2.0 (3e10681 build@2024-09-06T10:26:49)
+
rknn_server version: 2.3.0 (e80ac5c build@2024-11-07T12:52:53)
 
$ strings /usr/lib/librknnrt.so |grep 'librknnrt version:'
 
$ strings /usr/lib/librknnrt.so |grep 'librknnrt version:'
librknnrt version: 2.2.0 (c195366594@2024-09-14T12:18:56)
+
librknnrt version: 2.3.0 (c949ad889d@2024-11-07T11:35:33)
 
</syntaxhighlight>
 
</syntaxhighlight>
  
Line 62: Line 66:
 
sudo apt-get install -y python3-dev python3-numpy python3-opencv python3-pip
 
sudo apt-get install -y python3-dev python3-numpy python3-opencv python3-pip
 
cd ~/rknn-toolkit2
 
cd ~/rknn-toolkit2
pip3 install ./rknn-toolkit-lite2/packages/rknn_toolkit_lite2-2.2.0-cp39-cp39-linux_aarch64.whl -i https://pypi.tuna.tsinghua.edu.cn/simple/
+
pip3 install ./rknn-toolkit-lite2/packages/rknn_toolkit_lite2-2.3.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl -i https://pypi.tuna.tsinghua.edu.cn/simple/
 
</syntaxhighlight>
 
</syntaxhighlight>
  
Line 69: Line 73:
 
$ cd ~/rknn-toolkit2/rknn-toolkit-lite2/examples/resnet18/
 
$ cd ~/rknn-toolkit2/rknn-toolkit-lite2/examples/resnet18/
 
$ python3 test.py
 
$ python3 test.py
W rknn-toolkit-lite2 version: 2.2.0
+
W rknn-toolkit-lite2 version: 2.3.0
 
--> Load RKNN model
 
--> Load RKNN model
 
done
 
done
 
--> Init runtime environment
 
--> Init runtime environment
I RKNN: [06:50:39.456] RKNN Runtime Information, librknnrt version: 2.2.0 (c195366594@2024-09-14T12:18:56)
+
I RKNN: [06:29:43.602] RKNN Runtime Information, librknnrt version: 2.3.0 (c949ad889d@2024-11-07T11:35:33)
I RKNN: [06:50:39.456] RKNN Driver Information, version: 0.9.3
+
I RKNN: [06:29:43.602] RKNN Driver Information, version: 0.9.8
I RKNN: [06:50:39.456] RKNN Model Information, version: 6, toolkit version: 2.2.0(compiler version: 2.2.0 (c195366594@2024-09-14T12:24:14)), target: RKNPU v2, target platform: rk3588, framework name: PyTorch, framework layout: NCHW, model inference type: static_shape
+
I RKNN: [06:29:43.602] RKNN Model Information, version: 6, toolkit version: 2.3.0(compiler version: 2.3.0 (c949ad889d@2024-11-07T11:39:30)), target: RKNPU v2, target platform: rk3588, framework name: PyTorch, framework layout: NCHW, model inference type: static_shape
W RKNN: [06:50:39.474] query RKNN_QUERY_INPUT_DYNAMIC_RANGE error, rknn model is static shape type, please export rknn with dynamic_shapes
+
W RKNN: [06:29:43.617] query RKNN_QUERY_INPUT_DYNAMIC_RANGE error, rknn model is static shape type, please export rknn with dynamic_shapes
 
W Query dynamic range failed. Ret code: RKNN_ERR_MODEL_INVALID. (If it is a static shape RKNN model, please ignore the above warning message.)
 
W Query dynamic range failed. Ret code: RKNN_ERR_MODEL_INVALID. (If it is a static shape RKNN model, please ignore the above warning message.)
 
done
 
done
Line 98: Line 102:
 
     tk-dev libgdbm-compat-dev libreadline-dev
 
     tk-dev libgdbm-compat-dev libreadline-dev
  
wget https://www.python.org/ftp/python/3.9.16/Python-3.9.16.tar.xz
+
wget https://www.python.org/ftp/python/3.9.21/Python-3.9.21.tar.xz
tar -xvf Python-3.9.16.tar.xz
+
tar -xvf Python-3.9.21.tar.xz
cd Python-3.9.16/
+
cd Python-3.9.21/
 
./configure --enable-optimizations
 
./configure --enable-optimizations
 
make -j$(nproc)
 
make -j$(nproc)
Line 116: Line 120:
 
cd rknn-toolkit2
 
cd rknn-toolkit2
 
python3.9 -m pip install \
 
python3.9 -m pip install \
     ./rknn-toolkit-lite2/packages/rknn_toolkit_lite2-2.2.0-cp39-cp39-linux_aarch64.whl \
+
     ./rknn-toolkit-lite2/packages/rknn_toolkit_lite2-2.3.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl \
 
     -i https://pypi.tuna.tsinghua.edu.cn/simple/
 
     -i https://pypi.tuna.tsinghua.edu.cn/simple/
 
</syntaxhighlight>
 
</syntaxhighlight>
Line 127: Line 131:
 
output:
 
output:
 
<syntaxhighlight lang="text">
 
<syntaxhighlight lang="text">
W rknn-toolkit-lite2 version: 2.2.0
+
W rknn-toolkit-lite2 version: 2.3.0
 
--> Load RKNN model
 
--> Load RKNN model
 
done
 
done
 
--> Init runtime environment
 
--> Init runtime environment
I RKNN: [09:57:36.101] RKNN Runtime Information, librknnrt version: 2.2.0 (c195366594@2024-09-14T12:18:56)
+
I RKNN: [07:08:29.756] RKNN Runtime Information, librknnrt version: 2.3.0 (c949ad889d@2024-11-07T11:35:33)
I RKNN: [09:57:36.101] RKNN Driver Information, version: 0.9.3
+
I RKNN: [07:08:29.757] RKNN Driver Information, version: 0.9.8
I RKNN: [09:57:36.101] RKNN Model Information, version: 6, toolkit version: 2.2.0(compiler version: 2.2.0 (c195366594@2024-09-14T12:24:14)), target: RKNPU v2, target platform: rk3588, framework name: PyTorch, framework layout: NCHW, model inference type: static_shape
+
I RKNN: [07:08:29.762] RKNN Model Information, version: 6, toolkit version: 2.3.0(compiler version: 2.3.0 (c949ad889d@2024-11-07T11:39:30)), target: RKNPU v2, target platform: rk3588, framework name: PyTorch, framework layout: NCHW, model inference type: static_shape
W RKNN: [09:57:36.117] query RKNN_QUERY_INPUT_DYNAMIC_RANGE error, rknn model is static shape type, please export rknn with dynamic_shapes
+
W RKNN: [07:08:29.814] query RKNN_QUERY_INPUT_DYNAMIC_RANGE error, rknn model is static shape type, please export rknn with dynamic_shapes
 
W Query dynamic range failed. Ret code: RKNN_ERR_MODEL_INVALID. (If it is a static shape RKNN model, please ignore the above warning message.)
 
W Query dynamic range failed. Ret code: RKNN_ERR_MODEL_INVALID. (If it is a static shape RKNN model, please ignore the above warning message.)
 
done
 
done
Line 148: Line 152:
 
done
 
done
 
</syntaxhighlight>
 
</syntaxhighlight>
 +
==RKLLM and Large Language Models==
 +
===Model Conversion (PC Side)===
 +
The Hugging Face-format large language model needs to be converted into an RKLLM model to enable loading and usage on the Rockchip NPU platform. The conversion process must be performed on a PC, with Ubuntu 20.04 as the recommended operating system. A Docker container can also be used.
 +
====Installing RKLLM-Toolkit====
 +
<syntaxhighlight lang="text">
 +
sudo apt-get update
 +
sudo apt-get install -y python3-dev python3-numpy python3-opencv python3-pip
 +
cd ~
 +
git clone https://github.com/airockchip/rknn-llm
 +
pip3 install rknn-llm/rkllm-toolkit/packages/rkllm_toolkit-1.1.4-cp38-cp38-linux_x86_64.whl
 +
</syntaxhighlight>
 +
====Download and Convert the Model====
 +
Use the git clone command to download the required model from the Hugging Face repository. Update test.py to point to the downloaded model, then execute test.py to start the model conversion process:
 +
<syntaxhighlight lang="text">
 +
cd ~/rknn-llm/rkllm-toolkit/examples/
 +
git lfs install
 +
git clone https://huggingface.co/THUDM/chatglm3-6b
 +
sed -i 's|^modelpath\s*=.*|modelpath = "./chatglm3-6b"|' test.py
 +
sed -i 's|^dataset\s*=.*|dataset = None|' test.py
 +
python3 test.py
 +
</syntaxhighlight>
 +
The output information of the model conversion is as follows:
 +
<syntaxhighlight lang="text">
 +
ubuntu@20de568f4011:~/rknn-llm/rkllm-toolkit/examples$ python3 test.py
 +
INFO: rkllm-toolkit version: 1.1.4
 +
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
 +
Optimizing model: 100%|██████████████████████████████████████████████████████████████████████████████████| 24/24 [00:21<00:00,  1.13it/s]
 +
Building model: 100%|█████████████████████████████████████████████████████████████████████████████████| 343/343 [00:01<00:00, 296.74it/s]
 +
INFO: The token_id of eos is set to 151643
 +
INFO: The token_id of pad is set to 151643
 +
INFO: The token_id of bos is set to 151643
 +
Converting model: 100%|███████████████████████████████████████████████████████████████████████████| 291/291 [00:00<00:00, 2591385.27it/s]
 +
INFO: Exporting the model, please wait ....
 +
[=================================================>] 513/513 (100%)
 +
INFO: Model has been saved to ./qwen.rkllm!
 +
</syntaxhighlight>
 +
If successful, the file qwen.rkllm will be generated. Rename it to chatglm3-6b.rkllm and upload it to the development board:
 +
<syntaxhighlight lang="text">
 +
mv qwen.rkllm chatglm3-6b.rkllm
 +
scp chatglm3-6b.rkllm root@NanoPC-T6:/home/pi
 +
</syntaxhighlight>
 +
===Compile and run demo application (On Development Board)===
 +
<syntaxhighlight lang="text">
 +
sudo apt-get update
 +
sudo apt-get install -y gcc g++ make cmake
 +
 +
# get rknn-llm
 +
cd ~
 +
[ -d rknn-llm ] || git clone https://github.com/airockchip/rknn-llm
 +
cd rknn-llm/examples/rkllm_api_demo
 +
chmod +x ./build-linux.sh
 +
sed -i 's/^GCC_COMPILER_PATH=.*/GCC_COMPILER_PATH=aarch64-linux-gnu/' ./build-linux.sh
 +
./build-linux.sh
 +
 +
# run llm_demo
 +
cd ./build/build_linux_aarch64_Release/
 +
export LD_LIBRARY_PATH=~/rknn-llm/rkllm-runtime/Linux/librkllm_api/aarch64:$LD_LIBRARY_PATH
 +
ulimit -HSn 10240
 +
# max_new_tokens = 256, max_context_len = 512
 +
taskset f0 ./llm_demo ~/chatglm3-6b.rkllm 256 512
 +
</syntaxhighlight>
 +
Output:
 +
<syntaxhighlight lang="text">
 +
rkllm init start
 +
I rkllm: rkllm-runtime version: 1.1.4, rknpu driver version: 0.9.8, platform: RK3588
 +
 +
rkllm init success
 +
 +
**********************可输入以下问题对应序号获取回答/或自定义输入********************
 +
 +
[0] 把下面的现代文翻译成文言文: 到了春风和煦,阳光明媚的时候,湖面平静,没有惊涛骇浪,天色湖光相连,一片碧绿,广阔无际;沙洲上的鸥鸟,时而飞翔,时而停歇,美丽的鱼游来游去,岸上与小洲上的花草,青翠欲滴。
 +
[1] 以咏梅为题目,帮我写一首古诗,要求包含梅花、白雪等元素。
 +
[2] 上联: 江边惯看千帆过
 +
[3] 把这句话翻译成中文: Knowledge can be acquired from many sources. These include books, teachers and practical experience, and each has its own advantages. The knowledge we gain from books and formal education enables us to learn about things that we have no opportunity to experience in daily life. We can also develop our analytical skills and learn how to view and interpret the world around us in different ways. Furthermore, we can learn from the past by reading books. In this way, we won't repeat the mistakes of others and can build on their achievements.
 +
[4] 把这句话翻译成英文: RK3588是新一代高端处理器,具有高算力、低功耗、超强多媒体、丰富数据接口等特点
 +
 +
*************************************************************************
 +
 +
 +
user: 天空为什么是蓝色的?
 +
robot:
 +
蓝色是如何形成的?
 +
 +
天空为什么是蓝色的?这是因为大气层中的气体分子对太阳光的吸收和散射。当太阳光穿过大气层时,其中的可见光波(包括红、橙、黄、绿、蓝、靛、紫等颜色)与气体分子相互作用。其中,蓝色光的波长较短,能量较高。当蓝色光与气体分子相互作用时,它们被散射到各个方向,使得天空呈现出蓝色。
 +
 +
此外,太阳光中的蓝色光相对于其他颜色的光更容易被散射。这是因为在太阳光谱中,蓝色光的波长较短,频率较高。当光线穿过大气层时,空气中的气体分子和悬浮颗粒会对不同波长的光进行散射,而蓝色光的波长较短,能量较高,因此更容易被散射。这使得天空呈现出蓝色。
 +
 +
user:
 +
</syntaxhighlight>
 +
===Useful Resources===
 +
* https://github.com/airockchip/rknn-llm
 +
* https://github.com/wudingjian/rkllm_chat
 
==Doc==
 
==Doc==
 
https://github.com/rockchip-linux/rknpu2/tree/master/doc
 
https://github.com/rockchip-linux/rknpu2/tree/master/doc
Line 153: Line 249:
 
===View NPU Load===
 
===View NPU Load===
 
<syntaxhighlight lang="text">
 
<syntaxhighlight lang="text">
cat /sys/kernel/debug/rknpu/load
+
sudo cat /sys/kernel/debug/rknpu/load
 
</syntaxhighlight>
 
</syntaxhighlight>
 +
<!--
 
===Set NPU freq===
 
===Set NPU freq===
 
<syntaxhighlight lang="text">
 
<syntaxhighlight lang="text">
echo userspace > /sys/class/devfreq/fdab0000.npu/governor
+
sudo echo userspace > /sys/class/devfreq/fdab0000.npu/governor
echo 800000000 > /sys/class/devfreq/fdab0000.npu/min_freq
+
sudo echo 800000000 > /sys/class/devfreq/fdab0000.npu/min_freq
echo 1000000000 > /sys/class/devfreq/fdab0000.npu/max_freq
+
sudo echo 1000000000 > /sys/class/devfreq/fdab0000.npu/max_freq
 
</syntaxhighlight>
 
</syntaxhighlight>
 +
-->
 
===View NPU freq===
 
===View NPU freq===
 
<syntaxhighlight lang="text">
 
<syntaxhighlight lang="text">
 
cat /sys/class/devfreq/fdab0000.npu/cur_freq
 
cat /sys/class/devfreq/fdab0000.npu/cur_freq
 
</syntaxhighlight>
 
</syntaxhighlight>

Revision as of 17:13, 19 January 2025

查看中文

1 Earlier version RKNPU2 SDK

Link to → v1.5.2

2 How to test RKNN

2.1 OS

Tested on the following OS:

2.1.1 Debian11 (bullseye)

  • rk3588-sd-debian-bullseye-desktop-6.1-arm64-20250117.img.gz

2.1.2 Ubuntu20 (focal)

  • rk3588-sd-ubuntu-focal-desktop-6.1-arm64-20250117.img.gz

2.2 Check NPU driver version

$ sudo cat /sys/kernel/debug/rknpu/version
RKNPU driver: v0.9.8

2.3 install rknpu

cd ~
export GIT_SSL_NO_VERIFY=1
git clone https://github.com/airockchip/rknn-toolkit2.git
cd rknn-toolkit2/rknpu2
sudo cp ./runtime/Linux/librknn_api/aarch64/* /usr/lib
sudo cp ./runtime/Linux/rknn_server/aarch64/usr/bin/* /usr/bin/
sudo cp ./runtime/Linux/librknn_api/include/* /usr/include/

2.4 check rknn version

$ strings /usr/bin/rknn_server |grep 'build@'
2.3.0 (e80ac5c build@2024-11-07T12:52:53)
rknn_server version: 2.3.0 (e80ac5c build@2024-11-07T12:52:53)
$ strings /usr/lib/librknnrt.so |grep 'librknnrt version:'
librknnrt version: 2.3.0 (c949ad889d@2024-11-07T11:35:33)

2.5 run rknn_yolov5_demo

sudo apt-get update
sudo apt-get install -y gcc g++ make cmake
 
# fix broken link
cd ~/rknn-toolkit2/rknpu2/examples/3rdparty/mpp/Linux/aarch64
rm -f librockchip_mpp.so librockchip_mpp.so.1
ln -s librockchip_mpp.so.0 librockchip_mpp.so
ln -s librockchip_mpp.so.0 librockchip_mpp.so.1
 
cd ~/rknn-toolkit2/rknpu2/examples/rknn_yolov5_demo
chmod +x ./build-linux.sh
sudo ln -s /usr/bin/gcc /usr/bin/aarch64-gcc
sudo ln -s /usr/bin/g++ /usr/bin/aarch64-g++
export GCC_COMPILER=aarch64
./build-linux.sh -t rk3588 -a aarch64 -b Release
cd install/rknn_yolov5_demo_Linux
./rknn_yolov5_demo model/RK3588/yolov5s-640-640.rknn model/bus.jpg

Transfer the generated out.jpg to PC to view the result:

scp out.jpg xxx@YourIP:/tmp/

Rknn yolov5 demo out.jpg

2.6 install rknn_toolkit on debian11

2.6.1 install rknn_toolkit

sudo apt-get update
sudo apt-get install -y python3-dev python3-numpy python3-opencv python3-pip
cd ~/rknn-toolkit2
pip3 install ./rknn-toolkit-lite2/packages/rknn_toolkit_lite2-2.3.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl -i https://pypi.tuna.tsinghua.edu.cn/simple/

2.6.2 run python demo

$ cd ~/rknn-toolkit2/rknn-toolkit-lite2/examples/resnet18/
$ python3 test.py
W rknn-toolkit-lite2 version: 2.3.0
--> Load RKNN model
done
--> Init runtime environment
I RKNN: [06:29:43.602] RKNN Runtime Information, librknnrt version: 2.3.0 (c949ad889d@2024-11-07T11:35:33)
I RKNN: [06:29:43.602] RKNN Driver Information, version: 0.9.8
I RKNN: [06:29:43.602] RKNN Model Information, version: 6, toolkit version: 2.3.0(compiler version: 2.3.0 (c949ad889d@2024-11-07T11:39:30)), target: RKNPU v2, target platform: rk3588, framework name: PyTorch, framework layout: NCHW, model inference type: static_shape
W RKNN: [06:29:43.617] query RKNN_QUERY_INPUT_DYNAMIC_RANGE error, rknn model is static shape type, please export rknn with dynamic_shapes
W Query dynamic range failed. Ret code: RKNN_ERR_MODEL_INVALID. (If it is a static shape RKNN model, please ignore the above warning message.)
done
--> Running model
resnet18
-----TOP 5-----
[812] score:0.999680 class:"space shuttle"
[404] score:0.000249 class:"airliner"
[657] score:0.000013 class:"missile"
[466] score:0.000009 class:"bullet train, bullet"
[895] score:0.000008 class:"warplane, military plane"
 
done

2.7 install rknn_toolkit on ubuntu-focal

2.7.1 build python3.9 from source

sudo apt install build-essential libssl-dev libffi-dev software-properties-common \
    libbz2-dev libncurses-dev libncursesw5-dev libgdbm-dev liblzma-dev libsqlite3-dev \
    tk-dev libgdbm-compat-dev libreadline-dev
 
wget https://www.python.org/ftp/python/3.9.21/Python-3.9.21.tar.xz
tar -xvf Python-3.9.21.tar.xz
cd Python-3.9.21/
./configure --enable-optimizations
make -j$(nproc)
sudo make install

2.7.2 install rknn_toolkit

python3.9 -m pip install --upgrade pip
python3.9 -m pip install opencv-python
 
cd ~
[ -d rknn-toolkit2 ] || git clone https://github.com/airockchip/rknn-toolkit2.git \
    --depth 1 -b master
cd rknn-toolkit2
python3.9 -m pip install \
    ./rknn-toolkit-lite2/packages/rknn_toolkit_lite2-2.3.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl \
    -i https://pypi.tuna.tsinghua.edu.cn/simple/

2.7.3 run python demo on ubuntu-focal

cd ~/rknn-toolkit2/rknn-toolkit-lite2/examples/resnet18/
python3.9 test.py

output:

W rknn-toolkit-lite2 version: 2.3.0
--> Load RKNN model
done
--> Init runtime environment
I RKNN: [07:08:29.756] RKNN Runtime Information, librknnrt version: 2.3.0 (c949ad889d@2024-11-07T11:35:33)
I RKNN: [07:08:29.757] RKNN Driver Information, version: 0.9.8
I RKNN: [07:08:29.762] RKNN Model Information, version: 6, toolkit version: 2.3.0(compiler version: 2.3.0 (c949ad889d@2024-11-07T11:39:30)), target: RKNPU v2, target platform: rk3588, framework name: PyTorch, framework layout: NCHW, model inference type: static_shape
W RKNN: [07:08:29.814] query RKNN_QUERY_INPUT_DYNAMIC_RANGE error, rknn model is static shape type, please export rknn with dynamic_shapes
W Query dynamic range failed. Ret code: RKNN_ERR_MODEL_INVALID. (If it is a static shape RKNN model, please ignore the above warning message.)
done
--> Running model
resnet18
-----TOP 5-----
[812] score:0.999680 class:"space shuttle"
[404] score:0.000249 class:"airliner"
[657] score:0.000013 class:"missile"
[466] score:0.000009 class:"bullet train, bullet"
[895] score:0.000008 class:"warplane, military plane"
 
done

3 RKLLM and Large Language Models

3.1 Model Conversion (PC Side)

The Hugging Face-format large language model needs to be converted into an RKLLM model to enable loading and usage on the Rockchip NPU platform. The conversion process must be performed on a PC, with Ubuntu 20.04 as the recommended operating system. A Docker container can also be used.

3.1.1 Installing RKLLM-Toolkit

sudo apt-get update
sudo apt-get install -y python3-dev python3-numpy python3-opencv python3-pip
cd ~
git clone https://github.com/airockchip/rknn-llm
pip3 install rknn-llm/rkllm-toolkit/packages/rkllm_toolkit-1.1.4-cp38-cp38-linux_x86_64.whl

3.1.2 Download and Convert the Model

Use the git clone command to download the required model from the Hugging Face repository. Update test.py to point to the downloaded model, then execute test.py to start the model conversion process:

cd ~/rknn-llm/rkllm-toolkit/examples/
git lfs install
git clone https://huggingface.co/THUDM/chatglm3-6b
sed -i 's|^modelpath\s*=.*|modelpath = "./chatglm3-6b"|' test.py
sed -i 's|^dataset\s*=.*|dataset = None|' test.py
python3 test.py

The output information of the model conversion is as follows:

ubuntu@20de568f4011:~/rknn-llm/rkllm-toolkit/examples$ python3 test.py
INFO: rkllm-toolkit version: 1.1.4
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
Optimizing model: 100%|██████████████████████████████████████████████████████████████████████████████████| 24/24 [00:21<00:00,  1.13it/s]
Building model: 100%|█████████████████████████████████████████████████████████████████████████████████| 343/343 [00:01<00:00, 296.74it/s]
INFO: The token_id of eos is set to 151643
INFO: The token_id of pad is set to 151643
INFO: The token_id of bos is set to 151643
Converting model: 100%|███████████████████████████████████████████████████████████████████████████| 291/291 [00:00<00:00, 2591385.27it/s]
INFO: Exporting the model, please wait ....
[=================================================>] 513/513 (100%)
INFO: Model has been saved to ./qwen.rkllm!

If successful, the file qwen.rkllm will be generated. Rename it to chatglm3-6b.rkllm and upload it to the development board:

mv qwen.rkllm chatglm3-6b.rkllm
scp chatglm3-6b.rkllm root@NanoPC-T6:/home/pi

3.2 Compile and run demo application (On Development Board)

sudo apt-get update
sudo apt-get install -y gcc g++ make cmake
 
# get rknn-llm
cd ~
[ -d rknn-llm ] || git clone https://github.com/airockchip/rknn-llm
cd rknn-llm/examples/rkllm_api_demo
chmod +x ./build-linux.sh
sed -i 's/^GCC_COMPILER_PATH=.*/GCC_COMPILER_PATH=aarch64-linux-gnu/' ./build-linux.sh
./build-linux.sh
 
# run llm_demo
cd ./build/build_linux_aarch64_Release/
export LD_LIBRARY_PATH=~/rknn-llm/rkllm-runtime/Linux/librkllm_api/aarch64:$LD_LIBRARY_PATH
ulimit -HSn 10240
# max_new_tokens = 256, max_context_len = 512
taskset f0 ./llm_demo ~/chatglm3-6b.rkllm 256 512

Output:

rkllm init start
I rkllm: rkllm-runtime version: 1.1.4, rknpu driver version: 0.9.8, platform: RK3588
 
rkllm init success
 
**********************可输入以下问题对应序号获取回答/或自定义输入********************
 
[0] 把下面的现代文翻译成文言文: 到了春风和煦,阳光明媚的时候,湖面平静,没有惊涛骇浪,天色湖光相连,一片碧绿,广阔无际;沙洲上的鸥鸟,时而飞翔,时而停歇,美丽的鱼游来游去,岸上与小洲上的花草,青翠欲滴。
[1] 以咏梅为题目,帮我写一首古诗,要求包含梅花、白雪等元素。
[2] 上联: 江边惯看千帆过
[3] 把这句话翻译成中文: Knowledge can be acquired from many sources. These include books, teachers and practical experience, and each has its own advantages. The knowledge we gain from books and formal education enables us to learn about things that we have no opportunity to experience in daily life. We can also develop our analytical skills and learn how to view and interpret the world around us in different ways. Furthermore, we can learn from the past by reading books. In this way, we won't repeat the mistakes of others and can build on their achievements.
[4] 把这句话翻译成英文: RK3588是新一代高端处理器,具有高算力、低功耗、超强多媒体、丰富数据接口等特点
 
*************************************************************************
 
 
user: 天空为什么是蓝色的?
robot:
蓝色是如何形成的?
 
天空为什么是蓝色的?这是因为大气层中的气体分子对太阳光的吸收和散射。当太阳光穿过大气层时,其中的可见光波(包括红、橙、黄、绿、蓝、靛、紫等颜色)与气体分子相互作用。其中,蓝色光的波长较短,能量较高。当蓝色光与气体分子相互作用时,它们被散射到各个方向,使得天空呈现出蓝色。
 
此外,太阳光中的蓝色光相对于其他颜色的光更容易被散射。这是因为在太阳光谱中,蓝色光的波长较短,频率较高。当光线穿过大气层时,空气中的气体分子和悬浮颗粒会对不同波长的光进行散射,而蓝色光的波长较短,能量较高,因此更容易被散射。这使得天空呈现出蓝色。
 
user:

3.3 Useful Resources

4 Doc

https://github.com/rockchip-linux/rknpu2/tree/master/doc

5 Other

5.1 View NPU Load

sudo cat /sys/kernel/debug/rknpu/load

5.2 View NPU freq

cat /sys/class/devfreq/fdab0000.npu/cur_freq