Eyes Slide-Move:Classic-Inpainting fill hole and complete missing iris

I wanted to move only the eyes using Flux.1 schnell, but I couldn't get it to work well with prompts. I thought Flux.1 schnell-inpaint was broken, but I discovered that by using a donut mask, I can change the colors normally. So, I decided to create a guide image for that purpose. As it turns out, the guide image alone is quite usable. Also, I feel like I might be able to create rolling eyes and white eyes with Flux 1.schnell, even though they might not be supported? (Whether they are necessary or not is another matter).

• (1) Make "blank eyes" (remove irises) using classic-inpaint
• (2) complete missing iris hidden by eyelid using classic-inapaint
• (3) Move the iris to the desired (x, y) coordinates

Divide into an outer faces and an inner eyes

Mediapipe can detect face landmarks,it help divide into an outer faces and an inner eyes for editing

Make "blank eyes" (remove irises) using classic-inpaint

iris positions are detected by mediapipe

complete missing iris hidden by eyelid using classic-inapaint

mediapipe detecging iris size and location is not accurate,maybe need adjusting.

technically this inpainting is not so effect,almost same as just fill circle.

Move the iris to the desired (x, y) coordinates

To sharpen the blank eyes, I pasted an image of an iris over them. Unfortunately, this resulted in visible gaps around the iris. To correct these gaps, I applied a blur to the composite image and then pasted a sharpened image of the eye area excluding the iris, using a mask to blend the layers.

Limitations and Future Work:

The current implementation depends on Mediapipe, a pre-trained and non-trainable model, resulting in suboptimal performance for images that deviate significantly from a frontal view. To overcome this limitation, future work should focus on training a specialized landmark point detection model tailored to a wider range of head poses.

Related Spaces

• Mediapipe Change Eyes Direction I create all images by this space
• Flux.1 schnell use large size image or use small strength(0.5) value
• Flux.1 schnell inpaint - use donut option for better result
• classic-inapint - for try just classic-inpaint