Difference between revisions of "NPU"

From FriendlyELEC WiKi
Jump to: navigation, search
(updated by API)
(updated by API)
 
(8 intermediate revisions by the same user not shown)
Line 3: Line 3:
 
==Earlier version RKNPU2 SDK==
 
==Earlier version RKNPU2 SDK==
 
Link to → [[NPU_SDK1.5.2|v1.5.2]]
 
Link to → [[NPU_SDK1.5.2|v1.5.2]]
==How to test NPU==
+
==Check NPU driver version==
 +
<syntaxhighlight lang="text">
 +
$ sudo cat /sys/kernel/debug/rknpu/version
 +
RKNPU driver: v0.9.8
 +
</syntaxhighlight>
 +
==RKNN Quick Example==
 
===OS===
 
===OS===
 
Tested on the following OS:
 
Tested on the following OS:
 
====Debian11 (bullseye)====
 
====Debian11 (bullseye)====
* rk3588-sd-debian-bullseye-desktop-6.1-arm64-20241011.img.gz
+
* rk3588-sd-debian-bullseye-desktop-6.1-arm64-20250117.img.gz
 
====Ubuntu20 (focal)====
 
====Ubuntu20 (focal)====
* rk3588-sd-ubuntu-focal-desktop-6.1-arm64-20241011.img.gz
+
* rk3588-sd-ubuntu-focal-desktop-6.1-arm64-20250117.img.gz
===install rknpu===
+
===Installing RKNN Runtime===
 
<syntaxhighlight lang="text">
 
<syntaxhighlight lang="text">
 
cd ~
 
cd ~
 
export GIT_SSL_NO_VERIFY=1
 
export GIT_SSL_NO_VERIFY=1
git clone https://github.com/airockchip/rknn-toolkit2.git --depth 1 -b master
+
git clone https://github.com/airockchip/rknn-toolkit2.git
 
cd rknn-toolkit2/rknpu2
 
cd rknn-toolkit2/rknpu2
 
sudo cp ./runtime/Linux/librknn_api/aarch64/* /usr/lib
 
sudo cp ./runtime/Linux/librknn_api/aarch64/* /usr/lib
Line 20: Line 25:
 
sudo cp ./runtime/Linux/librknn_api/include/* /usr/include/
 
sudo cp ./runtime/Linux/librknn_api/include/* /usr/include/
 
</syntaxhighlight>
 
</syntaxhighlight>
 
+
===Check rknn version===
===check rknn version===
+
 
<syntaxhighlight lang="text">
 
<syntaxhighlight lang="text">
 
$ strings /usr/bin/rknn_server |grep 'build@'
 
$ strings /usr/bin/rknn_server |grep 'build@'
2.2.0 (3e10681 build@2024-09-06T10:26:49)
+
2.3.0 (e80ac5c build@2024-11-07T12:52:53)
rknn_server version: 2.2.0 (3e10681 build@2024-09-06T10:26:49)
+
rknn_server version: 2.3.0 (e80ac5c build@2024-11-07T12:52:53)
 
$ strings /usr/lib/librknnrt.so |grep 'librknnrt version:'
 
$ strings /usr/lib/librknnrt.so |grep 'librknnrt version:'
librknnrt version: 2.2.0 (c195366594@2024-09-14T12:18:56)
+
librknnrt version: 2.3.0 (c949ad889d@2024-11-07T11:35:33)
 
</syntaxhighlight>
 
</syntaxhighlight>
  
===run rknn_yolov5_demo===
+
===C++ Example: Build and run the YOLOv5 demo===
 
<syntaxhighlight lang="text">
 
<syntaxhighlight lang="text">
 
sudo apt-get update
 
sudo apt-get update
Line 56: Line 60:
 
[[File:Rknn_yolov5_demo_out.jpg|640px]]
 
[[File:Rknn_yolov5_demo_out.jpg|640px]]
  
===install rknn_toolkit on debian11===
+
===Python Example: Install and Test RKNN Toolkit Lite2 on Debian11===
====install rknn_toolkit====
+
====Install RKNN Toolkit Lite2====
 
<syntaxhighlight lang="text">
 
<syntaxhighlight lang="text">
 
sudo apt-get update
 
sudo apt-get update
 
sudo apt-get install -y python3-dev python3-numpy python3-opencv python3-pip
 
sudo apt-get install -y python3-dev python3-numpy python3-opencv python3-pip
 
cd ~/rknn-toolkit2
 
cd ~/rknn-toolkit2
pip3 install ./rknn-toolkit-lite2/packages/rknn_toolkit_lite2-2.2.0-cp39-cp39-linux_aarch64.whl -i https://pypi.tuna.tsinghua.edu.cn/simple/
+
pip3 install ./rknn-toolkit-lite2/packages/rknn_toolkit_lite2-2.3.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl -i https://pypi.tuna.tsinghua.edu.cn/simple/
 
</syntaxhighlight>
 
</syntaxhighlight>
  
====run python demo====
+
====Run the python example====
 
<syntaxhighlight lang="text">
 
<syntaxhighlight lang="text">
 
$ cd ~/rknn-toolkit2/rknn-toolkit-lite2/examples/resnet18/
 
$ cd ~/rknn-toolkit2/rknn-toolkit-lite2/examples/resnet18/
 
$ python3 test.py
 
$ python3 test.py
W rknn-toolkit-lite2 version: 2.2.0
+
W rknn-toolkit-lite2 version: 2.3.0
 
--> Load RKNN model
 
--> Load RKNN model
 
done
 
done
 
--> Init runtime environment
 
--> Init runtime environment
I RKNN: [06:50:39.456] RKNN Runtime Information, librknnrt version: 2.2.0 (c195366594@2024-09-14T12:18:56)
+
I RKNN: [06:29:43.602] RKNN Runtime Information, librknnrt version: 2.3.0 (c949ad889d@2024-11-07T11:35:33)
I RKNN: [06:50:39.456] RKNN Driver Information, version: 0.9.3
+
I RKNN: [06:29:43.602] RKNN Driver Information, version: 0.9.8
I RKNN: [06:50:39.456] RKNN Model Information, version: 6, toolkit version: 2.2.0(compiler version: 2.2.0 (c195366594@2024-09-14T12:24:14)), target: RKNPU v2, target platform: rk3588, framework name: PyTorch, framework layout: NCHW, model inference type: static_shape
+
I RKNN: [06:29:43.602] RKNN Model Information, version: 6, toolkit version: 2.3.0(compiler version: 2.3.0 (c949ad889d@2024-11-07T11:39:30)), target: RKNPU v2, target platform: rk3588, framework name: PyTorch, framework layout: NCHW, model inference type: static_shape
W RKNN: [06:50:39.474] query RKNN_QUERY_INPUT_DYNAMIC_RANGE error, rknn model is static shape type, please export rknn with dynamic_shapes
+
W RKNN: [06:29:43.617] query RKNN_QUERY_INPUT_DYNAMIC_RANGE error, rknn model is static shape type, please export rknn with dynamic_shapes
 
W Query dynamic range failed. Ret code: RKNN_ERR_MODEL_INVALID. (If it is a static shape RKNN model, please ignore the above warning message.)
 
W Query dynamic range failed. Ret code: RKNN_ERR_MODEL_INVALID. (If it is a static shape RKNN model, please ignore the above warning message.)
 
done
 
done
Line 91: Line 95:
 
</syntaxhighlight>
 
</syntaxhighlight>
  
===install rknn_toolkit on ubuntu-focal===
+
===Python Example: Install and Test RKNN Toolkit Lite2 on Ubuntu 20.04 (Focal)===
====build python3.9 from source====
+
====Compile and install python3.9====
 
<syntaxhighlight lang="text">
 
<syntaxhighlight lang="text">
 
sudo apt install build-essential libssl-dev libffi-dev software-properties-common \
 
sudo apt install build-essential libssl-dev libffi-dev software-properties-common \
Line 98: Line 102:
 
     tk-dev libgdbm-compat-dev libreadline-dev
 
     tk-dev libgdbm-compat-dev libreadline-dev
  
wget https://www.python.org/ftp/python/3.9.16/Python-3.9.16.tar.xz
+
wget https://www.python.org/ftp/python/3.9.21/Python-3.9.21.tar.xz
tar -xvf Python-3.9.16.tar.xz
+
tar -xvf Python-3.9.21.tar.xz
cd Python-3.9.16/
+
cd Python-3.9.21/
 
./configure --enable-optimizations
 
./configure --enable-optimizations
 
make -j$(nproc)
 
make -j$(nproc)
Line 106: Line 110:
 
</syntaxhighlight>
 
</syntaxhighlight>
  
====install rknn_toolkit====
+
====Install RKNN Toolkit Lite2====
 
<syntaxhighlight lang="text">
 
<syntaxhighlight lang="text">
 
python3.9 -m pip install --upgrade pip
 
python3.9 -m pip install --upgrade pip
Line 116: Line 120:
 
cd rknn-toolkit2
 
cd rknn-toolkit2
 
python3.9 -m pip install \
 
python3.9 -m pip install \
     ./rknn-toolkit-lite2/packages/rknn_toolkit_lite2-2.2.0-cp39-cp39-linux_aarch64.whl \
+
     ./rknn-toolkit-lite2/packages/rknn_toolkit_lite2-2.3.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl \
 
     -i https://pypi.tuna.tsinghua.edu.cn/simple/
 
     -i https://pypi.tuna.tsinghua.edu.cn/simple/
 
</syntaxhighlight>
 
</syntaxhighlight>
  
====run python demo on ubuntu-focal====
+
====Run the python example====
 
<syntaxhighlight lang="text">
 
<syntaxhighlight lang="text">
 
cd ~/rknn-toolkit2/rknn-toolkit-lite2/examples/resnet18/
 
cd ~/rknn-toolkit2/rknn-toolkit-lite2/examples/resnet18/
Line 127: Line 131:
 
output:
 
output:
 
<syntaxhighlight lang="text">
 
<syntaxhighlight lang="text">
W rknn-toolkit-lite2 version: 2.2.0
+
W rknn-toolkit-lite2 version: 2.3.0
 
--> Load RKNN model
 
--> Load RKNN model
 
done
 
done
 
--> Init runtime environment
 
--> Init runtime environment
I RKNN: [09:57:36.101] RKNN Runtime Information, librknnrt version: 2.2.0 (c195366594@2024-09-14T12:18:56)
+
I RKNN: [07:08:29.756] RKNN Runtime Information, librknnrt version: 2.3.0 (c949ad889d@2024-11-07T11:35:33)
I RKNN: [09:57:36.101] RKNN Driver Information, version: 0.9.3
+
I RKNN: [07:08:29.757] RKNN Driver Information, version: 0.9.8
I RKNN: [09:57:36.101] RKNN Model Information, version: 6, toolkit version: 2.2.0(compiler version: 2.2.0 (c195366594@2024-09-14T12:24:14)), target: RKNPU v2, target platform: rk3588, framework name: PyTorch, framework layout: NCHW, model inference type: static_shape
+
I RKNN: [07:08:29.762] RKNN Model Information, version: 6, toolkit version: 2.3.0(compiler version: 2.3.0 (c949ad889d@2024-11-07T11:39:30)), target: RKNPU v2, target platform: rk3588, framework name: PyTorch, framework layout: NCHW, model inference type: static_shape
W RKNN: [09:57:36.117] query RKNN_QUERY_INPUT_DYNAMIC_RANGE error, rknn model is static shape type, please export rknn with dynamic_shapes
+
W RKNN: [07:08:29.814] query RKNN_QUERY_INPUT_DYNAMIC_RANGE error, rknn model is static shape type, please export rknn with dynamic_shapes
 
W Query dynamic range failed. Ret code: RKNN_ERR_MODEL_INVALID. (If it is a static shape RKNN model, please ignore the above warning message.)
 
W Query dynamic range failed. Ret code: RKNN_ERR_MODEL_INVALID. (If it is a static shape RKNN model, please ignore the above warning message.)
 
done
 
done
Line 148: Line 152:
 
done
 
done
 
</syntaxhighlight>
 
</syntaxhighlight>
 +
==RKLLM and Large Language Models==
 +
===Model Conversion (PC Side)===
 +
The Hugging Face-format large language model needs to be converted into an RKLLM model to enable loading and usage on the Rockchip NPU platform. The conversion process must be performed on a PC, with Ubuntu 20.04 as the recommended operating system. A Docker container can also be used.
 +
====Installing RKLLM-Toolkit====
 +
<syntaxhighlight lang="text">
 +
sudo apt-get update
 +
sudo apt-get install -y python3-dev python3-numpy python3-opencv python3-pip
 +
cd ~
 +
git clone https://github.com/airockchip/rknn-llm
 +
pip3 install rknn-llm/rkllm-toolkit/packages/rkllm_toolkit-1.1.4-cp38-cp38-linux_x86_64.whl
 +
</syntaxhighlight>
 +
====Download and Convert the Model====
 +
Use the git clone command to download the required model from the Hugging Face repository. Update test.py to point to the downloaded model, then execute test.py to start the model conversion process:
 +
<syntaxhighlight lang="text">
 +
cd ~/rknn-llm/rkllm-toolkit/examples/
 +
git lfs install
 +
git clone https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B
 +
sed -i 's|^modelpath\s*=.*|modelpath = "./DeepSeek-R1-Distill-Qwen-1.5B"|' test.py
 +
sed -i 's|^dataset\s*=.*|dataset = None|' test.py
 +
python3 test.py
 +
</syntaxhighlight>
 +
The output information of the model conversion is as follows:
 +
<syntaxhighlight lang="text">
 +
ubuntu@20de568f4011:~/rknn-llm/rkllm-toolkit/examples$ python3 test.py
 +
INFO: rkllm-toolkit version: 1.1.4
 +
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
 +
Optimizing model: 100%|██████████████████████████████████████████████████████████████████████████████████████| 28/28 [01:33<00:00,  3.35s/it]
 +
Building model: 100%|█████████████████████████████████████████████████████████████████████████████████████| 399/399 [00:03<00:00, 102.43it/s]
 +
WARNING: The bos token has two ids: 151646 and 151643, please ensure that the bos token ids in config.json and tokenizer_config.json are consistent!
 +
INFO: The token_id of bos is set to 151646
 +
INFO: The token_id of eos is set to 151643
 +
INFO: The token_id of pad is set to 151643
 +
Converting model: 100%|███████████████████████████████████████████████████████████████████████████████| 339/339 [00:00<00:00, 5642337.52it/s]
 +
INFO: Exporting the model, please wait ....
 +
[=================================================>] 597/597 (100%)
 +
INFO: Model has been saved to ./qwen.rkllm!
 +
</syntaxhighlight>
 +
If successful, the file qwen.rkllm will be generated. Rename it to DeepSeek-R1-Distill-Qwen-1.5B.rkllm and upload it to the development board:
 +
<syntaxhighlight lang="text">
 +
mv qwen.rkllm DeepSeek-R1-Distill-Qwen-1.5B.rkllm
 +
scp DeepSeek-R1-Distill-Qwen-1.5B.rkllm root@NanoPC-T6:/home/pi
 +
</syntaxhighlight>
 +
===Compile and run demo application (On Development Board)===
 +
<syntaxhighlight lang="text">
 +
sudo apt-get update
 +
sudo apt-get install -y gcc g++ make cmake
 +
 +
# get rknn-llm
 +
cd ~
 +
[ -d rknn-llm ] || git clone https://github.com/airockchip/rknn-llm
 +
cd rknn-llm/examples/rkllm_api_demo
 +
chmod +x ./build-linux.sh
 +
sed -i 's/^GCC_COMPILER_PATH=.*/GCC_COMPILER_PATH=aarch64-linux-gnu/' ./build-linux.sh
 +
./build-linux.sh
 +
 +
# run llm_demo
 +
cd ./build/build_linux_aarch64_Release/
 +
export LD_LIBRARY_PATH=~/rknn-llm/rkllm-runtime/Linux/librkllm_api/aarch64:$LD_LIBRARY_PATH
 +
ulimit -HSn 10240
 +
# max_new_tokens = 512, max_context_len = 1024
 +
taskset f0 ./llm_demo ~/DeepSeek-R1-Distill-Qwen-1.5B.rkllm 512 1024
 +
</syntaxhighlight>
 +
Output:
 +
<syntaxhighlight lang="text">
 +
rkllm init start
 +
I rkllm: rkllm-runtime version: 1.1.4, rknpu driver version: 0.9.8, platform: RK3588
 +
 +
rkllm init success
 +
 +
**********************可输入以下问题对应序号获取回答/或自定义输入********************
 +
 +
[0] 把下面的现代文翻译成文言文: 到了春风和煦,阳光明媚的时候,湖面平静,没有惊涛骇浪,天色湖光相连,一片碧绿,广阔无际;沙洲上的鸥鸟,时而飞翔,时而停歇,美丽的鱼游来游去,岸上与小洲上的花草,青翠欲滴。
 +
[1] 以咏梅为题目,帮我写一首古诗,要求包含梅花、白雪等元素。
 +
[2] 上联: 江边惯看千帆过
 +
[3] 把这句话翻译成中文: Knowledge can be acquired from many sources. These include books, teachers and practical experience, and each has its own advantages. The knowledge we gain from books and formal education enables us to learn about things that we have no opportunity to experience in daily life. We can also develop our analytical skills and learn how to view and interpret the world around us in different ways. Furthermore, we can learn from the past by reading books. In this way, we won't repeat the mistakes of others and can build on their achievements.
 +
[4] 把这句话翻译成英文: RK3588是新一代高端处理器,具有高算力、低功耗、超强多媒体、丰富数据接口等特点
 +
 +
*************************************************************************
 +
 +
 +
user: 天空为什么是蓝色的?
 +
robot: 这个问题看起来很有趣,但我不太确定。我想知道天空的颜色是什么原因导致的,以及是否有其他因素影响了它的颜色。
 +
首先,我想到的是太阳本身的颜色。太阳看起来像一个红色的东西,对吧?那是不是和天空的颜色有关呢?
 +
然后,我还记得以前学过一点关于大气层的知识,说大气层是透明的,但有一定的折射作用。这可能影响了我们看到的天空颜色。
 +
另外,我还听说过光的反射和折射的一些定律,比如费马原理,可能也与天空的颜色有关。
 +
不过,我还不太确定这些因素是如何相互作用的,以及它们最终导致了天空为什么会是蓝色的。
 +
 +
也许我可以先从太阳本身的颜色开始分析。太阳看起来红色,是因为它发出的光大部分在红光区,而蓝光和绿光被大气层吸收或散射,所以我们看到的是红色。那如果太阳本身是蓝色的话,会不会影响天空的颜色呢?
 +
另外,我还记得有人说过,天空的颜色可能与地球上的天气状况有关,比如阳光是否直射、云的类型等等。
 +
不过,这些信息我还不太确定,需要进一步查证。
 +
 +
总的来说,我觉得天空的颜色是由多种因素共同作用的结果,包括太阳的颜色、大气层的折射和反射、光的反射定律以及天气状况等。但具体来说,为什么天空会是蓝色呢?这可能涉及到光的折射和反射现象,以及地球表面的光线分布。
 +
</think>
 +
 +
天空呈现蓝色的原因主要归因于光的折射和反射现象,以及地球表面光线分布的影响。以下是详细的解释:
 +
 +
1. **太阳的颜色**:太阳本身看起来红色,这是因为太阳发出的大部分光在红光区,而蓝光和绿光被大气层吸收或散射,所以我们看到的是红色。
 +
 +
2. **大气层的作用**:
 +
  - 大气层是透明的,但具有折射作用。当阳光穿过大气层时,部分光线会被折射,导致天空呈现蓝色。
 +
  - 这种现象称为“大气折射”或“海市蜃楼效应”。不同颜色的光在不同介质中的折射率不同,导致它们在空气中传播时发生偏折。
 +
 +
3. **光的反射和折射**:
 +
  - 光线在空气中的传播遵循费马原理,即光线会尽可能地快速到达目标。这包括反射和折射。
 +
  - 太阳光穿过大气层后,在地球表面反射,而部分光被折射进入天空,
 +
 +
user:
 +
</syntaxhighlight>
 +
===Useful Resources===
 +
* https://github.com/airockchip/rknn-llm
 +
* https://github.com/wudingjian/rkllm_chat
 
==Doc==
 
==Doc==
 
https://github.com/rockchip-linux/rknpu2/tree/master/doc
 
https://github.com/rockchip-linux/rknpu2/tree/master/doc
Line 153: Line 268:
 
===View NPU Load===
 
===View NPU Load===
 
<syntaxhighlight lang="text">
 
<syntaxhighlight lang="text">
cat /sys/kernel/debug/rknpu/load
+
sudo cat /sys/kernel/debug/rknpu/load
 
</syntaxhighlight>
 
</syntaxhighlight>
 
===Set NPU freq===
 
===Set NPU freq===
 
<syntaxhighlight lang="text">
 
<syntaxhighlight lang="text">
echo userspace > /sys/class/devfreq/fdab0000.npu/governor
+
echo userspace | sudo tee /sys/class/devfreq/fdab0000.npu/governor > /dev/null
echo 800000000 > /sys/class/devfreq/fdab0000.npu/min_freq
+
echo 800000000 | sudo tee /sys/class/devfreq/fdab0000.npu/min_freq > /dev/null
echo 1000000000 > /sys/class/devfreq/fdab0000.npu/max_freq
+
echo 1000000000 | sudo tee /sys/class/devfreq/fdab0000.npu/max_freq > /dev/null
 
</syntaxhighlight>
 
</syntaxhighlight>
 
===View NPU freq===
 
===View NPU freq===

Latest revision as of 06:02, 7 February 2025

查看中文

1 Earlier version RKNPU2 SDK

Link to → v1.5.2

2 Check NPU driver version

$ sudo cat /sys/kernel/debug/rknpu/version
RKNPU driver: v0.9.8

3 RKNN Quick Example

3.1 OS

Tested on the following OS:

3.1.1 Debian11 (bullseye)

  • rk3588-sd-debian-bullseye-desktop-6.1-arm64-20250117.img.gz

3.1.2 Ubuntu20 (focal)

  • rk3588-sd-ubuntu-focal-desktop-6.1-arm64-20250117.img.gz

3.2 Installing RKNN Runtime

cd ~
export GIT_SSL_NO_VERIFY=1
git clone https://github.com/airockchip/rknn-toolkit2.git
cd rknn-toolkit2/rknpu2
sudo cp ./runtime/Linux/librknn_api/aarch64/* /usr/lib
sudo cp ./runtime/Linux/rknn_server/aarch64/usr/bin/* /usr/bin/
sudo cp ./runtime/Linux/librknn_api/include/* /usr/include/

3.3 Check rknn version

$ strings /usr/bin/rknn_server |grep 'build@'
2.3.0 (e80ac5c build@2024-11-07T12:52:53)
rknn_server version: 2.3.0 (e80ac5c build@2024-11-07T12:52:53)
$ strings /usr/lib/librknnrt.so |grep 'librknnrt version:'
librknnrt version: 2.3.0 (c949ad889d@2024-11-07T11:35:33)

3.4 C++ Example: Build and run the YOLOv5 demo

sudo apt-get update
sudo apt-get install -y gcc g++ make cmake
 
# fix broken link
cd ~/rknn-toolkit2/rknpu2/examples/3rdparty/mpp/Linux/aarch64
rm -f librockchip_mpp.so librockchip_mpp.so.1
ln -s librockchip_mpp.so.0 librockchip_mpp.so
ln -s librockchip_mpp.so.0 librockchip_mpp.so.1
 
cd ~/rknn-toolkit2/rknpu2/examples/rknn_yolov5_demo
chmod +x ./build-linux.sh
sudo ln -s /usr/bin/gcc /usr/bin/aarch64-gcc
sudo ln -s /usr/bin/g++ /usr/bin/aarch64-g++
export GCC_COMPILER=aarch64
./build-linux.sh -t rk3588 -a aarch64 -b Release
cd install/rknn_yolov5_demo_Linux
./rknn_yolov5_demo model/RK3588/yolov5s-640-640.rknn model/bus.jpg

Transfer the generated out.jpg to PC to view the result:

scp out.jpg xxx@YourIP:/tmp/

Rknn yolov5 demo out.jpg

3.5 Python Example: Install and Test RKNN Toolkit Lite2 on Debian11

3.5.1 Install RKNN Toolkit Lite2

sudo apt-get update
sudo apt-get install -y python3-dev python3-numpy python3-opencv python3-pip
cd ~/rknn-toolkit2
pip3 install ./rknn-toolkit-lite2/packages/rknn_toolkit_lite2-2.3.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl -i https://pypi.tuna.tsinghua.edu.cn/simple/

3.5.2 Run the python example

$ cd ~/rknn-toolkit2/rknn-toolkit-lite2/examples/resnet18/
$ python3 test.py
W rknn-toolkit-lite2 version: 2.3.0
--> Load RKNN model
done
--> Init runtime environment
I RKNN: [06:29:43.602] RKNN Runtime Information, librknnrt version: 2.3.0 (c949ad889d@2024-11-07T11:35:33)
I RKNN: [06:29:43.602] RKNN Driver Information, version: 0.9.8
I RKNN: [06:29:43.602] RKNN Model Information, version: 6, toolkit version: 2.3.0(compiler version: 2.3.0 (c949ad889d@2024-11-07T11:39:30)), target: RKNPU v2, target platform: rk3588, framework name: PyTorch, framework layout: NCHW, model inference type: static_shape
W RKNN: [06:29:43.617] query RKNN_QUERY_INPUT_DYNAMIC_RANGE error, rknn model is static shape type, please export rknn with dynamic_shapes
W Query dynamic range failed. Ret code: RKNN_ERR_MODEL_INVALID. (If it is a static shape RKNN model, please ignore the above warning message.)
done
--> Running model
resnet18
-----TOP 5-----
[812] score:0.999680 class:"space shuttle"
[404] score:0.000249 class:"airliner"
[657] score:0.000013 class:"missile"
[466] score:0.000009 class:"bullet train, bullet"
[895] score:0.000008 class:"warplane, military plane"
 
done

3.6 Python Example: Install and Test RKNN Toolkit Lite2 on Ubuntu 20.04 (Focal)

3.6.1 Compile and install python3.9

sudo apt install build-essential libssl-dev libffi-dev software-properties-common \
    libbz2-dev libncurses-dev libncursesw5-dev libgdbm-dev liblzma-dev libsqlite3-dev \
    tk-dev libgdbm-compat-dev libreadline-dev
 
wget https://www.python.org/ftp/python/3.9.21/Python-3.9.21.tar.xz
tar -xvf Python-3.9.21.tar.xz
cd Python-3.9.21/
./configure --enable-optimizations
make -j$(nproc)
sudo make install

3.6.2 Install RKNN Toolkit Lite2

python3.9 -m pip install --upgrade pip
python3.9 -m pip install opencv-python
 
cd ~
[ -d rknn-toolkit2 ] || git clone https://github.com/airockchip/rknn-toolkit2.git \
    --depth 1 -b master
cd rknn-toolkit2
python3.9 -m pip install \
    ./rknn-toolkit-lite2/packages/rknn_toolkit_lite2-2.3.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl \
    -i https://pypi.tuna.tsinghua.edu.cn/simple/

3.6.3 Run the python example

cd ~/rknn-toolkit2/rknn-toolkit-lite2/examples/resnet18/
python3.9 test.py

output:

W rknn-toolkit-lite2 version: 2.3.0
--> Load RKNN model
done
--> Init runtime environment
I RKNN: [07:08:29.756] RKNN Runtime Information, librknnrt version: 2.3.0 (c949ad889d@2024-11-07T11:35:33)
I RKNN: [07:08:29.757] RKNN Driver Information, version: 0.9.8
I RKNN: [07:08:29.762] RKNN Model Information, version: 6, toolkit version: 2.3.0(compiler version: 2.3.0 (c949ad889d@2024-11-07T11:39:30)), target: RKNPU v2, target platform: rk3588, framework name: PyTorch, framework layout: NCHW, model inference type: static_shape
W RKNN: [07:08:29.814] query RKNN_QUERY_INPUT_DYNAMIC_RANGE error, rknn model is static shape type, please export rknn with dynamic_shapes
W Query dynamic range failed. Ret code: RKNN_ERR_MODEL_INVALID. (If it is a static shape RKNN model, please ignore the above warning message.)
done
--> Running model
resnet18
-----TOP 5-----
[812] score:0.999680 class:"space shuttle"
[404] score:0.000249 class:"airliner"
[657] score:0.000013 class:"missile"
[466] score:0.000009 class:"bullet train, bullet"
[895] score:0.000008 class:"warplane, military plane"
 
done

4 RKLLM and Large Language Models

4.1 Model Conversion (PC Side)

The Hugging Face-format large language model needs to be converted into an RKLLM model to enable loading and usage on the Rockchip NPU platform. The conversion process must be performed on a PC, with Ubuntu 20.04 as the recommended operating system. A Docker container can also be used.

4.1.1 Installing RKLLM-Toolkit

sudo apt-get update
sudo apt-get install -y python3-dev python3-numpy python3-opencv python3-pip
cd ~
git clone https://github.com/airockchip/rknn-llm
pip3 install rknn-llm/rkllm-toolkit/packages/rkllm_toolkit-1.1.4-cp38-cp38-linux_x86_64.whl

4.1.2 Download and Convert the Model

Use the git clone command to download the required model from the Hugging Face repository. Update test.py to point to the downloaded model, then execute test.py to start the model conversion process:

cd ~/rknn-llm/rkllm-toolkit/examples/
git lfs install
git clone https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B
sed -i 's|^modelpath\s*=.*|modelpath = "./DeepSeek-R1-Distill-Qwen-1.5B"|' test.py
sed -i 's|^dataset\s*=.*|dataset = None|' test.py
python3 test.py

The output information of the model conversion is as follows:

ubuntu@20de568f4011:~/rknn-llm/rkllm-toolkit/examples$ python3 test.py
INFO: rkllm-toolkit version: 1.1.4
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
Optimizing model: 100%|██████████████████████████████████████████████████████████████████████████████████████| 28/28 [01:33<00:00,  3.35s/it]
Building model: 100%|█████████████████████████████████████████████████████████████████████████████████████| 399/399 [00:03<00:00, 102.43it/s]
WARNING: The bos token has two ids: 151646 and 151643, please ensure that the bos token ids in config.json and tokenizer_config.json are consistent!
INFO: The token_id of bos is set to 151646
INFO: The token_id of eos is set to 151643
INFO: The token_id of pad is set to 151643
Converting model: 100%|███████████████████████████████████████████████████████████████████████████████| 339/339 [00:00<00:00, 5642337.52it/s]
INFO: Exporting the model, please wait ....
[=================================================>] 597/597 (100%)
INFO: Model has been saved to ./qwen.rkllm!

If successful, the file qwen.rkllm will be generated. Rename it to DeepSeek-R1-Distill-Qwen-1.5B.rkllm and upload it to the development board:

mv qwen.rkllm DeepSeek-R1-Distill-Qwen-1.5B.rkllm
scp DeepSeek-R1-Distill-Qwen-1.5B.rkllm root@NanoPC-T6:/home/pi

4.2 Compile and run demo application (On Development Board)

sudo apt-get update
sudo apt-get install -y gcc g++ make cmake
 
# get rknn-llm
cd ~
[ -d rknn-llm ] || git clone https://github.com/airockchip/rknn-llm
cd rknn-llm/examples/rkllm_api_demo
chmod +x ./build-linux.sh
sed -i 's/^GCC_COMPILER_PATH=.*/GCC_COMPILER_PATH=aarch64-linux-gnu/' ./build-linux.sh
./build-linux.sh
 
# run llm_demo
cd ./build/build_linux_aarch64_Release/
export LD_LIBRARY_PATH=~/rknn-llm/rkllm-runtime/Linux/librkllm_api/aarch64:$LD_LIBRARY_PATH
ulimit -HSn 10240
# max_new_tokens = 512, max_context_len = 1024
taskset f0 ./llm_demo ~/DeepSeek-R1-Distill-Qwen-1.5B.rkllm 512 1024

Output:

rkllm init start
I rkllm: rkllm-runtime version: 1.1.4, rknpu driver version: 0.9.8, platform: RK3588
 
rkllm init success
 
**********************可输入以下问题对应序号获取回答/或自定义输入********************
 
[0] 把下面的现代文翻译成文言文: 到了春风和煦,阳光明媚的时候,湖面平静,没有惊涛骇浪,天色湖光相连,一片碧绿,广阔无际;沙洲上的鸥鸟,时而飞翔,时而停歇,美丽的鱼游来游去,岸上与小洲上的花草,青翠欲滴。
[1] 以咏梅为题目,帮我写一首古诗,要求包含梅花、白雪等元素。
[2] 上联: 江边惯看千帆过
[3] 把这句话翻译成中文: Knowledge can be acquired from many sources. These include books, teachers and practical experience, and each has its own advantages. The knowledge we gain from books and formal education enables us to learn about things that we have no opportunity to experience in daily life. We can also develop our analytical skills and learn how to view and interpret the world around us in different ways. Furthermore, we can learn from the past by reading books. In this way, we won't repeat the mistakes of others and can build on their achievements.
[4] 把这句话翻译成英文: RK3588是新一代高端处理器,具有高算力、低功耗、超强多媒体、丰富数据接口等特点
 
*************************************************************************
 
 
user: 天空为什么是蓝色的?
robot: 这个问题看起来很有趣,但我不太确定。我想知道天空的颜色是什么原因导致的,以及是否有其他因素影响了它的颜色。
首先,我想到的是太阳本身的颜色。太阳看起来像一个红色的东西,对吧?那是不是和天空的颜色有关呢?
然后,我还记得以前学过一点关于大气层的知识,说大气层是透明的,但有一定的折射作用。这可能影响了我们看到的天空颜色。
另外,我还听说过光的反射和折射的一些定律,比如费马原理,可能也与天空的颜色有关。
不过,我还不太确定这些因素是如何相互作用的,以及它们最终导致了天空为什么会是蓝色的。
 
也许我可以先从太阳本身的颜色开始分析。太阳看起来红色,是因为它发出的光大部分在红光区,而蓝光和绿光被大气层吸收或散射,所以我们看到的是红色。那如果太阳本身是蓝色的话,会不会影响天空的颜色呢?
另外,我还记得有人说过,天空的颜色可能与地球上的天气状况有关,比如阳光是否直射、云的类型等等。
不过,这些信息我还不太确定,需要进一步查证。
 
总的来说,我觉得天空的颜色是由多种因素共同作用的结果,包括太阳的颜色、大气层的折射和反射、光的反射定律以及天气状况等。但具体来说,为什么天空会是蓝色呢?这可能涉及到光的折射和反射现象,以及地球表面的光线分布。
</think>
 
天空呈现蓝色的原因主要归因于光的折射和反射现象,以及地球表面光线分布的影响。以下是详细的解释:
 
1. **太阳的颜色**:太阳本身看起来红色,这是因为太阳发出的大部分光在红光区,而蓝光和绿光被大气层吸收或散射,所以我们看到的是红色。
 
2. **大气层的作用**:
   - 大气层是透明的,但具有折射作用。当阳光穿过大气层时,部分光线会被折射,导致天空呈现蓝色。
   - 这种现象称为“大气折射”或“海市蜃楼效应”。不同颜色的光在不同介质中的折射率不同,导致它们在空气中传播时发生偏折。
 
3. **光的反射和折射**:
   - 光线在空气中的传播遵循费马原理,即光线会尽可能地快速到达目标。这包括反射和折射。
   - 太阳光穿过大气层后,在地球表面反射,而部分光被折射进入天空,
 
user:

4.3 Useful Resources

5 Doc

https://github.com/rockchip-linux/rknpu2/tree/master/doc

6 Other

6.1 View NPU Load

sudo cat /sys/kernel/debug/rknpu/load

6.2 Set NPU freq

echo userspace | sudo tee /sys/class/devfreq/fdab0000.npu/governor > /dev/null
echo 800000000 | sudo tee /sys/class/devfreq/fdab0000.npu/min_freq > /dev/null
echo 1000000000 | sudo tee /sys/class/devfreq/fdab0000.npu/max_freq > /dev/null

6.3 View NPU freq

cat /sys/class/devfreq/fdab0000.npu/cur_freq