Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 28 additions & 0 deletions .github/workflows/face-recognition.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
name: face-recognition

on:
push:
branches: [master]
pull_request:
paths:
- rust/face-recognition/**
- .github/workflows/face-recognition.yml

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

jobs:
rust-face-recognition:
runs-on: ubuntu-24.04
container: ghcr.io/dfinity/icp-dev-env-rust:1.0.0
env:
ICP_CLI_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
- name: Deploy and test
working-directory: rust/face-recognition
run: |
icp network start -d
icp deploy
make test
46 changes: 0 additions & 46 deletions .github/workflows/rust-face-recognition-example.yaml

This file was deleted.

5 changes: 0 additions & 5 deletions rust/face-recognition/.cargo/config.toml

This file was deleted.

8 changes: 4 additions & 4 deletions rust/face-recognition/.gitignore
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
.dfx/
build/
node_modules/
dist/
Expand All @@ -8,6 +7,7 @@ _MACOSX
target/
*.old.did
.idea
src/backend/assets/version-RFB-320.onnx
src/backend/assets/facerec.onnx
.env
backend/assets/version-RFB-320.onnx
backend/assets/facerec.onnx
frontend/src/bindings/
.icp/cache/
2 changes: 1 addition & 1 deletion rust/face-recognition/Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
[workspace]
members = ["src/backend"]
members = ["backend"]
resolver = "2"
34 changes: 34 additions & 0 deletions rust/face-recognition/Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
.PHONY: test upload-models

test:
@echo "=== Test 1: clear_face_detection_model_bytes returns unit ==="
@result=$$(icp canister call backend clear_face_detection_model_bytes '()') && \
echo "$$result" && \
echo "PASS" || (echo "FAIL" && exit 1)

@echo "=== Test 2: clear_face_recognition_model_bytes returns unit ==="
@result=$$(icp canister call backend clear_face_recognition_model_bytes '()') && \
echo "$$result" && \
echo "PASS" || (echo "FAIL" && exit 1)

@echo "=== Test 3: append_face_detection_model_bytes accepts bytes ==="
@result=$$(icp canister call backend append_face_detection_model_bytes '(blob "\00\01\02")') && \
echo "$$result" && \
echo "PASS" || (echo "FAIL" && exit 1)

@echo "=== Test 4: run_detection returns Err when model is not set up ==="
@result=$$(icp canister call --query backend run_detection '()') && \
echo "$$result" && \
echo "$$result" | grep -q 'Err' && \
echo "PASS" || (echo "FAIL" && exit 1)

# Upload ONNX model files to the canister incrementally.
# Requires: ic-file-uploader (cargo install ic-file-uploader)
# First download the models: ./download-face-detection-model.sh
# and obtain face-recognition.onnx (see README), then run: make upload-models
upload-models:
icp canister call backend clear_face_detection_model_bytes '()'
icp canister call backend clear_face_recognition_model_bytes '()'
ic-file-uploader backend append_face_detection_model_bytes version-RFB-320.onnx
ic-file-uploader backend append_face_recognition_model_bytes face-recognition.onnx
icp canister call backend setup_models '()'
104 changes: 51 additions & 53 deletions rust/face-recognition/README.md
Original file line number Diff line number Diff line change
@@ -1,96 +1,94 @@
# ICP face recognition

This is an ICP smart contract runs face detection and face recognition of user's photo that can be uploaded either from a camera or a local file.
This is an ICP smart contract that runs face detection and face recognition on user photos that can be uploaded either from a camera or a local file.

The smart contract consists of two canisters:

- The backend canister embeds the [the Tract ONNX inference engine](https://github.com/sonos/tract) with two ONNX models. One model is used to detect a face in the photo and return its bounding box. Another model is used for computing face embeddings.
- The frontend canister contains the Web assets such as HTML, JS, CSS that are served to the browser.
- The **backend canister** embeds the [Tract ONNX inference engine](https://github.com/sonos/tract) with two ONNX models. One model is used to detect a face in the photo and return its bounding box. Another model is used for computing face embeddings.
- The **frontend canister** contains the Web assets such as HTML, JS, and CSS that are served to the browser.

# Models
## Models

The smart contract uses two models: one for detecting the face and another for recognizing the face.

## Face detection
Since the models are large, they cannot be embedded into the Wasm binary of the smart contract. Instead they must be uploaded separately after deployment.

A face detection model finds the bounding box of a face in the image.
You can download [Ultraface](https://github.com/onnx/models/tree/main/validated/vision/body_analysis/ultraface) - ultra-lightweight face detection model - [[here](https://github.com/onnx/models/blob/bec48b6a70e5e9042c0badbaafefe4454e072d08/validated/vision/body_analysis/ultraface/models/version-RFB-320.onnx)].
### Face detection

Alternatively, you can run
A face detection model finds the bounding box of a face in the image.
You can download [Ultraface](https://github.com/onnx/models/tree/main/validated/vision/body_analysis/ultraface) — ultra-lightweight face detection model — by running:

```
```bash
./download-face-detection-model.sh
```

## Face recognition
### Face recognition

A face recognition model computes a vector embedding of an image with a face.
You can obtain a pretrained model from [facenet-pytorch](https://github.com/timesler/facenet-pytorch) as follows.

- #### Step 1: Install `python` and `pip`: https://packaging.python.org/en/latest/tutorials/installing-packages/.
1. Install `python` and `pip`: https://packaging.python.org/en/latest/tutorials/installing-packages/

- #### Step 2: Install `facenet-pytorch` and `torch`:
2. Install `facenet-pytorch`, `torch`, and `onnx`:

```
pip install facenet-pytorch
pip install torch
pip install onnx
```
```bash
pip install facenet-pytorch
pip install torch
pip install onnx
```

- #### Step 3: Export ONNX model. Start a python shell and run the following commands or create a python file and run it:
3. Export the ONNX model. Start a Python shell and run:

```
import torch
import facenet_pytorch
resnet = facenet_pytorch.InceptionResnetV1(pretrained='vggface2').eval()
input = torch.randn(1, 3, 160, 160)
torch.onnx.export(resnet, input, "face-recognition.onnx", verbose=False, opset_version=11)
```
```python
import torch
import facenet_pytorch
resnet = facenet_pytorch.InceptionResnetV1(pretrained='vggface2').eval()
input = torch.randn(1, 3, 160, 160)
torch.onnx.export(resnet, input, "face-recognition.onnx", verbose=False, opset_version=11)
```

- #### Step 4: This should produce `face-recognition.onnx`. Copy the file to the root of this repository.
This produces `face-recognition.onnx`. Copy the file to the root of this repository.

## Prerequisites
## Build and deploy from the command line

- [x] Install the [IC
SDK](https://internetcomputer.org/docs/current/developer-docs/getting-started/install). For local testing, `dfx >= 0.22.0` is required.
- [x] Clone the example dapp project: `git clone https://github.com/dfinity/examples`
- [x] Install `wasi2ic`: Follow the steps in https://github.com/wasm-forge/wasi2ic and make sure that `wasi2ic` binary is in your `$PATH`.
- [x] Install `wasm-opt`: `cargo install wasm-opt`
### Prerequisites

## Build the application
- Node.js
- icp-cli: `npm install -g @icp-sdk/icp-cli @icp-sdk/ic-wasm`

```
dfx start --background
dfx deploy
### Install

```bash
git clone https://github.com/dfinity/examples
cd examples/rust/face-recognition
```

If the deployment is successful, the it will show the `frontend` URL.
Open that URL in browser to interact with the smart contract.
### Deploy and test

## Chunk uploading of models
```bash
icp network start -d
icp deploy
make test
icp network stop
```

Since the models are large, they cannot be embedded into the Wasm binary of the smart contract.
Instead they should be uploaded separately.
Run `npm run dev` from the `frontend/` directory for hot reload during frontend development.

[DecideAI](https://decideai.xyz/) implemented a tool for incremental uploading of models: https://github.com/modclub-app/ic-file-uploader/tree/main.
### Upload the ONNX models

You can install the tool with
After deploying, upload the ONNX models to the canister using [ic-file-uploader](https://github.com/modclub-app/ic-file-uploader/tree/main):

```
```bash
cargo install ic-file-uploader
make upload-models
```

Afterwards, execute the `upload-models-to-canister.sh` script, which runs the following commands:

```
dfx canister call backend clear_face_detection_model_bytes
dfx canister call backend clear_face_recognition_model_bytes
ic-file-uploader backend append_face_detection_model_bytes version-RFB-320.onnx
ic-file-uploader backend append_face_recognition_model_bytes face-recognition.onnx
dfx canister call backend setup_models
```
Once the models are uploaded, open the frontend URL in your browser to interact with the smart contract.

## Credits

Thanks to [DecideAI](https://decideai.xyz/) for discussions and providing [ic-file-uploader](https://github.com/modclub-app/ic-file-uploader/tree/main).

## Security considerations and best practices

See the [ICP security best practices](https://docs.internetcomputer.org/guides/security/overview).
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
[package]
edition = "2021"
name = "backend"
version = "1.1.0"
edition = "2021"

[lib]
crate-type = ["cdylib"]
Expand All @@ -19,4 +19,3 @@ prost = "0.11.0"
prost-types = "0.11.0"
serde = { version = "1.0", features = ["derive"] }
tract-onnx = { git = "https://github.com/sonos/tract", rev = "2a2914ac29390cc08963301c9f3d437b52dd321a" }

8 changes: 0 additions & 8 deletions rust/face-recognition/build.sh

This file was deleted.

37 changes: 0 additions & 37 deletions rust/face-recognition/dfx.json

This file was deleted.

Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
<link rel="icon" href="favicon.ico" />
<link type="text/css" rel="stylesheet" href="main.css" />
<link href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.0.0-beta3/css/all.min.css" rel="stylesheet">
<script type="module" src="./src/index.js"></script>
</head>

<body>
Expand Down Expand Up @@ -41,4 +42,4 @@ <h1>Onchain ICP <img id="inline-logo" src="logo_small.png"></img> face recogniti
</div>
</body>

</html>
</html>
17 changes: 17 additions & 0 deletions rust/face-recognition/frontend/package.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
{
"name": "frontend",
"private": true,
"type": "module",
"scripts": {
"prebuild": "npm i --include=dev",
"build": "vite build",
"dev": "vite"
},
"dependencies": {
"@icp-sdk/core": "~5.0.0"
},
"devDependencies": {
"@icp-sdk/bindgen": "~0.2.2",
"vite": "5.4.11"
}
}
Loading
Loading