LHM 3D Characters: Turning a Single Image into an Animatable 3D Model

I recently came across something really cool called LHM, which stands for Large Animatable Human Reconstruction Model. This technology can take a single picture of a person and transform it into a fully animatable 3D model. You can make the model dance, move, or perform any action you want—just by providing a reference video.

What is LHM 3D?

LHM 3D (Large Animatable Human Reconstruction Model) is an AI-powered tool that converts a single image into a 3D model and animates it using reference videos. It preserves facial features, clothing details, and can animate both real and cartoon characters.

LHM Overview:

Detail	Description
Name	LHM - Large Animatable Human Reconstruction Model
Purpose	Converts a single image into a fully animatable 3D model
Paper	LHM Paper on arXiv
GitHub Repository	LHM GitHub Code
Official Website	LHM Official Website
HuggingFace Demo	Try LHM on Hugging Face

How LHM Works?

Creating a 3D Model from a Single Image

The process is simple:

• You input a single image of a person (or even a 2D character).
• LHM generates a 3D model of that person while preserving facial and clothing details.

Here are some examples:

Example 1: Realistic Photo Conversion

• Input: An image of a person (left side).
• Output: A 3D model of that person.
• Animation: If you provide a reference video, LHM maps the motion from the video onto the 3D character.

Example 2: Preserving Fine Details

• Input: An image of a woman in a red t-shirt.
• Output: A 3D model that accurately captures her facial features, clothing details (like the logo on her hat and the swan on her shirt), and even hand movements when animated.

Example 3: 2D Characters Work Too

• Input: A 2D cartoon character.
• Output: A 3D version of that character.
• Animation: The same dance video can be applied to make the 3D character move realistically.

Example 4: Fictional Characters

• Input: A 2D ninja illustration.
• Output: A 3D model of the ninja.
• Animation: The ninja can be made to dance or perform actions based on a reference video.

Strengths of LHM

• Speed: Generates 3D avatars in just a few seconds.
• Detail Retention: Maintains facial accuracy and clothing details extremely well.
• Versatility: Works with both realistic photos and stylized 2D characters.

Limitations

While impressive, the 3D models aren’t perfect:

• Some flaws are noticeable around the edges of the body and fingers.
• The realism level isn’t photorealistic, but it’s one of the best available right now.

Beyond Realistic Photos

It can also convert 2D cartoon characters into 3D models. For example:

• A fictional 2D ninja character can be converted into a 3D model.
• The model can then perform actions like dancing using motion mapping from a video.

Quality and Speed

While the generated 3D models may have some minor imperfections, such as rough edges and finger inaccuracies, the AI produces results much faster than other tools. Within seconds, a high-quality 3D avatar is ready for animation.

Key Advantages:

• Speed: Generates 3D models in just a few seconds.
• Accuracy: Preserves facial features and clothing details.
• Versatility: Works with both real and animated images.

How to Try LHM Yourself

Hugging Face Demo

The developers have released a Hugging Face demo for public testing. Here’s how to use it:

1. Input Section:

   • Upload a reference image of the character you want to convert.
   • Upload a reference video to define the motion.

2. Generate:

   • Click the generate button to create the animated 3D model.

Note:

• Since this generates a video, you can’t use free Hugging Face credits.
• You’ll need to purchase GPU credits to run it.

Local Installation via GitHub

For those who prefer running it locally:

• A GitHub repository is available with full setup instructions.
• The model is lightweight—the largest version has only 1 billion parameters, making it feasible for local use.

Environment Setup

1. Clone the repository:

   git clone git@github.com:aigc3d/LHM.git
   cd LHM

2. Windows Installation:

   • Set Up a Virtual Environment:

     • Open Command Prompt (CMD), navigate to the project folder, and run:

       python -m venv lhm_env
       lhm_env\Scripts\activate

   • Install Dependencies:

     install_cu121.bat
     python ./app.py

   • CUDA Setup:

     • For CUDA 11.8:

       pip install rembg
       sh ./install_cu118.sh

     • For CUDA 12.1:

       sh ./install_cu121.sh

   • The installation has been tested with Python 3.10, CUDA 11.8, or CUDA 12.1.

   • Alternatively, you can install dependencies step by step by following INSTALL.md.

Model Weights

• The model will be downloaded automatically if you do not download it yourself.

• Model Details:

  Model	Training Data	BH-T Layers	Link	Inference Time
  LHM-0.5B	5K Synthetic Data	5	OSS	2.01 s
  LHM-0.5B	300K Videos + 5K Synthetic Data	5	OSS	2.01 s
  LHM-0.7B	300K Videos + 5K Synthetic Data	10	OSS	4.13 s
  LHM-1.0B	300K Videos + 5K Synthetic Data	15	OSS	6.57 s

• Download Prior Model Weights:

  wget https://virutalbuy-public.oss-cn-hangzhou.aliyuncs.com/share/aigc3d/data/for_lingteng/LHM/LHM-0.5B.tar
  tar -xvf LHM-0.5B.tar 
  wget https://virutalbuy-public.oss-cn-hangzhou.aliyuncs.com/share/aigc3d/data/for_lingteng/LHM/LHM-1B.tar
  tar -xvf LHM-1B.tar 

• Download Prior Model Weights:

  wget https://virutalbuy-public.oss-cn-hangzhou.aliyuncs.com/share/aigc3d/data/for_lingteng/LHM/LHM_prior_model.tar
  tar -xvf LHM_prior_model.tar 

Data Motion Preparation

• We provide the test motion examples and will update the processing scripts ASAP.

• Download Motion Video:

  wget https://virutalbuy-public.oss-cn-hangzhou.aliyuncs.com/share/aigc3d/data/for_lingteng/LHM/motion_video.tar
  tar -xvf ./motion_video.tar 

Final Thoughts

This AI is super useful for creating 3D avatars quickly and efficiently. If you're working with real people or fictional characters, LHM delivers impressive results with minimal input.