Skip to content

deep2universe/ChromePoseDream

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

15 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

DEPRECATED Repository

Use YouTube-Motion-Tracking

YouTube pose dream

Teaser
Some examples from this Video source

For more examples and the current progress see #visualizeYouTube playlist

Logo
The logo was designed by a 3 year old girl.

YouTube pose dream is a Chrome AI extension to visualize videos. (Soon also available for Firefox.)

When you're old enough, you may remember Winamp. There is the possibility to visualize music.
However, this extension makes similar with videos. The human pose in the video is used as input for the visualization.
In the process, the existing video is transformed into a psychedelic work of art. But this depends on the video you are watching.

The goal of this project is for anyone, with or without programming experience, to be able to visualize YouTube.
In addition, animations should be interchangeable between users.

Pose estimation is done with TensorFlow.js
Particle animation is done with Proton

This project is a Google Chrome browser extension. That's why you need this browser to try it out.

Table of Contents

  1. How it works
  2. Features
    1. Fun with lines
      1. Skeleton
      2. Skeleton three times
      3. Skeleton five times
      4. Puppets player
      5. Spider web
    2. Replace head with image
      1. Cat
      2. Smiley
      3. Sun
      4. Monkey
      5. Anonymous
    3. Show particle animation
      1. Hand power balls
      2. Two head balls
      3. Right hand line
      4. Nose gravity
      5. Nose supernova
      6. Hands track from border
      7. Upper body glow
      8. Glow painting
      9. Particle painting
      10. Particle painting with random drift
      11. Comet thrower
      12. Body glow
      13. Burning Man
  3. Installation
    1. Download this repository
    2. Open Chrome extensions
    3. Enable developer mode
    4. Load extension
    5. Check extension installation
    6. Check Chrome settings
    7. Pin the extension
  4. Usage
  5. Uninstall
  6. Build
  7. Further development
  8. License

How it works.

The Chrome extension becomes active, when you watch a YouTube video.
howItWorks

The extension use MoveNet as model.
Visual representation of the keypoints:
keypoints
Image from tensorflow.js pose-detection model
We use these keypoints to place animations or static content in the video.

Features

After the installation you can select the following visualizations via the extension menu.
You can change the visualization while the video is playing.
popup menu

Or you can change the visualization directly in the video player.
This also works in full screen mode.
With the new button the popup can be displayed or disabled.
player popup

The video Fatboy Slim ft. Bootsy Collins - Weapon Of Choice Official 4k Video was used for all screenshots.

original image

Fun with lines

Skeleton

This is TensorFlow.js in action.
MoveNet is used for the detection.
Use: all keypoints
pose detection

Skeleton 3 times

Use: all keypoints
skeleton3times

Skeleton 5 times

Use: all keypoints
skeleton5times

Puppets player

Use: left_wrist, right_wrist, left_shoulder, right_shoulder, left_ankle, right_ankle, nose
puppets player

Spider web

Use: all keypoints
spider web

Replace head with image

The eyes are used as a reference for displaying the image.
All use left_eye, right_eye

Cat

cat face

Smiley

smiley face

Sun

sun as face

Monkey

monkey face

Anonymous

anonymous

Show particle animation

Hand power balls

Every hand gets one energy ball.
Use: left_wrist, right_wrist
hand power balls

Two head balls

Two energy balls circle around the head.
Use: nose
two head balls

Right hand line

Only the right hand gets a particle animation.
Use: right_wrist
Right hand line

Nose gravity

A gravitational field is created around the nose.
Use: nose
Nose gravity

Nose supernova

Supernovas are constantly exploding on the nose.
Use: nose
Nose supernova

Hands track from border

Use: left_wrist, right_wrist
The yellow ray is connected to the right hand.
The red ray is connected to the left hand.
If the hands are too far away from the edge, the connection is interrupted.
Hands track from border

Upper body glow

Use: left_wrist, right_wrist, left_elbow, right_elbow, left_shoulder, right_shoulder
upper body glow

Glow painting

Use: left_wrist, right_wrist
Glow painting

Particle painting

Use: left_wrist, right_wrist
Particle painting

Particle painting with random drift

Use: left_wrist, right_wrist
Particle painting with random drift

Comet thrower

Use: left_wrist, right_wrist
Comet thrower

Body glow

Use: left_wrist, right_wrist, left_elbow, right_elbow, left_shoulder, right_shoulder, left_ankle, right_ankle, left_hip, right_hip
Body glow

Burning Man

Use: left_wrist, right_wrist, left_elbow, right_elbow, 2*left_shoulder, 2*right_shoulder, 2*left_ankle, 2*right_ankle, left_hip, right_hip, left_ear, right_ear, nose
Burning Man

Installation

Download this repository

git clone https://github.com/deep2universe/ChromePoseDream.git

Open Chrome extensions

Open this URL

chrome://extensions

Enable developer mode

In the upper right corner you have to activate the developer mode.
developer mode

Load extension

Click this button to load the extension.
Then select the dist Folder from this repository.
load

Check extension installation

You should now see the following entry.
extension

Check Chrome settings

Go to this URL

chrome://settings/accessibility

In the System settings make sure to enable this:
hardware chrome settings

Pin the extension

In the upper right corner click this to pin the extension.
pin

Usage

Open YouTube

Watch a video and have fun.

Uninstall

Just go to the chrome://extensions page and disable or delete the extension.
This will be possible later via the popup/settings page.

Build

All the code is in the src directory.
You need PARCEL for the build.

# install PARCEL
npm install -g parcel-bundler

# install dependencies
npm install

# build dist folder
npm run build

Further development

The following is still on the TODO list:

  • Support for other video platforms (e.g. Vimeo)
  • Support for multi-pose detection.
  • Internationalization
  • Export of the pose detection keypoints for other applications.
  • Use pose detection keypoints to feed GAN or VAE model and display result in webpage.
  • It should be possible to deactivate the extension. Currently, you must do this in chrome://extensions
  • Fill settings page with content (default animation, disable extension, language settings etc.)
  • Publish to Chrome extension store.
  • Clean up code.
  • Add support for three.proton to enable 3D particles.
  • Check content.js video event listener, - sometimes you have to reload the page to start pose dream.
  • Add more particle animations.
  • Create Firefox extension and publish to Firefox Browser ADD-ONS

This is an example of how this project setup could be extended in the future.
Currently, the focus of the project is on exploring the existing architectural possibilities. workflowExtended With the help of artificial intelligence, many other enhancements are feasible. For example, the acoustic content of the video is not viewed and could also be used.

If you want to support, pull requests are always welcome.
If you find any bugs please report them.

License

Apache License 2.0