yeemachine /kalidokit
Blendshape and kinematics calculator for Mediapipe/Tensorflow.js Face, Eyes, Pose, and Finger tracking models.
★ 5,730⑂ 721👁 78⑃ 21● 0▤ 0
+1
Daily
5,730
stars
721
forks
78
watchers
21
prs
0
issues
0
releases
README
[![NPM Package][npm]][npm-url]
[![NPM Bundle Size][minimized-size]][npm-url]
[![jsDelivr hits (npm)][js-delivr]][js-delivr-url]
[![NPM Downloads][npm-downloads]][npmtrends-url]
[![Website][website]][website-url]
[![Discord Shield][discord]][discord-url]
| :exclamation: KalidoKit is now officially deprecated and solutions will be directly integrated into MediaPipe :exclamation:|
|--------------------------------------|
Face, Pose, and Hand Tracking Calculator
Kalidokit is a blendshape and kinematics solver for Mediapipe/Tensorflow.js face, eyes, pose, and hand tracking models, compatible with Facemesh, Blazepose, Handpose, and Holistic. It takes predicted 3D landmarks and calculates simple euler rotations and blendshape face values.
As the core to Vtuber web apps, Kalidoface and Kalidoface 3D, KalidoKit is designed specifically for rigging 3D VRM models and Live2D avatars!
NOTE: This is a JS library meant for developers using Mediapipe pretrained models and not a complete app by itself. Please visit Kalidoface for the full vtuber experience 😉!
Install
Via NPM
npm install kalidokit
js
import as Kalidokit from "kalidokit";
// or only import the class you need
import { Face, Pose, Hand } from "kalidokit";
Via CDN
js
Methods
Kalidokit is composed of 3 classes for Face, Pose, and Hand calculations. They accept landmark outputs from models like Facemesh, Blazepose, Handpose, and Holistic.
js
// Accepts an array(468 or 478 with iris tracking) of vectors
Kalidokit.Face.solve(facelandmarkArray, {
runtime: "tfjs", // mediapipe or tfjs
video: HTMLVideoElement,
imageSize: { height: 0, width: 0 },
smoothBlink: false, // smooth left and right eye blink delays
blinkSettings: [0.25, 0.75], // adjust upper and lower bound blink sensitivity
});
// Accepts arrays(33) of Pose keypoints and 3D Pose keypoints
Kalidokit.Pose.solve(poseWorld3DArray, poseLandmarkArray, {
runtime: "tfjs", // mediapipe or tfjs
video: HTMLVideoElement,
imageSize: { height: 0, width: 0 },
enableLegs: true,
});
// Accepts array(21) of hand landmark vectors; specify 'Right' or 'Left' side
Kalidokit.Hand.solve(handLandmarkArray, "Right");
// Using exported classes directly
Face.solve(facelandmarkArray);
Pose.solve(poseWorld3DArray, poseLandmarkArray);
Hand.solve(handLandmarkArray, "Right");
Additional Utils
js
// Stabilizes left/right blink delays + wink by providing blenshapes and head rotation
Kalidokit.Face.stabilizeBlink(
{ r: 0, l: 1 }, // left and right eye blendshape values
headRotationY, // head rotation in radians
{
noWink = false, // disables winking
maxRot = 0.5 // max head rotation in radians before interpolating obscured eyes
});
// The internal vector math class
Kalidokit.Vector();
VRM and Live2D Sample Code
Quick-start your Vtuber app with these simple remixable examples.
VRM Glitch Template
Face, full-body, and hand tracking in under 350 lines of javascript. This dem