MakiPan commited on
Commit
c2802dd
·
1 Parent(s): bd81c24

Update app.py

Browse files
Files changed (1) hide show
  1. app.py +14 -10
app.py CHANGED
@@ -242,8 +242,8 @@ with gr.Blocks(theme='gradio/soft') as demo:
242
  with gr.Column():
243
  output_image = gr.Gallery(label='Output Image', show_label=False, elem_id="gallery").style(grid=2, height='auto')
244
 
245
- gr.Examples(
246
- examples=[
247
  [
248
  "a woman is making an ok sign in front of a painting",
249
  "longbody, lowres, bad anatomy, bad hands, missing fingers, extra digit, fewer digits, cropped, worst quality, low quality",
@@ -276,8 +276,8 @@ with gr.Blocks(theme='gradio/soft') as demo:
276
  cache_examples=True,
277
  )
278
 
279
- inputs = [prompt_input, negative_prompt, input_image, model_type]
280
- submit_btn.click(fn=infer, inputs=inputs, outputs=[output_image])
281
 
282
  gr.Markdown("""
283
  <center><h1>Summary</h1></center>
@@ -287,9 +287,11 @@ As Stable diffusion and other diffusion models are notoriously poor at generatin
287
  We opted to use the [HAnd Gesture Recognition Image Dataset](https://github.com/hukenovs/hagrid) (HaGRID) and [MediaPipe's Hand Landmarker](https://developers.google.com/mediapipe/solutions/vision/hand_landmarker) to train a control net that could potentially be used independently or as an in-painting tool.
288
  <br>
289
  To preprocess the data there were three options we considered:
 
 
 
290
  <br>
291
- > * The first was to use Mediapipes built-in draw landmarks function. This was an obvious first choice however we noticed with low training steps that the model couldn't easily distinguish handedness and would often generate the wrong hand for the conditioning image.<center>
292
- <br>
293
  <table><tr>
294
  <td>
295
  <p align="center" style="padding: 10px">
@@ -308,8 +310,9 @@ To preprocess the data there were three options we considered:
308
  </tr></table>
309
  </center>
310
  <br>
311
- > * To counter this issue we changed the palm landmark colors with the intention to keep the color similar in order to learn that they provide similar information, but different to make the model know which hands were left or right.<center>
312
- <br>
 
313
  <table><tr>
314
  <td>
315
  <p align="center" style="padding: 10px">
@@ -328,9 +331,10 @@ To preprocess the data there were three options we considered:
328
  </tr></table>
329
  </center>
330
  <br>
331
- > * The last option was to use [MediaPipe Holistic](https://ai.googleblog.com/2020/12/mediapipe-holistic-simultaneous-face.html) to provide pose face and hand landmarks to the ControlNet. This method was promising in theory, however, the HaGRID dataset was not suitable for this method as the Holistic model performs poorly with partial body and obscurely cropped images.
 
332
  <br>
333
- We anecdotally determined that when trained at lower steps the encoded hand model performed better than the standard MediaPipe model due to implied handedness. We theorize that with a larger dataset of more full-body hand and pose classifications, Holistic landmarks will provide the best images in the future however for the moment the hand encoded model performs best. """)
334
 
335
 
336
  demo.launch()
 
242
  with gr.Column():
243
  output_image = gr.Gallery(label='Output Image', show_label=False, elem_id="gallery").style(grid=2, height='auto')
244
 
245
+ gr.Examples(
246
+ examples=[
247
  [
248
  "a woman is making an ok sign in front of a painting",
249
  "longbody, lowres, bad anatomy, bad hands, missing fingers, extra digit, fewer digits, cropped, worst quality, low quality",
 
276
  cache_examples=True,
277
  )
278
 
279
+ inputs = [prompt_input, negative_prompt, input_image, model_type]
280
+ submit_btn.click(fn=infer, inputs=inputs, outputs=[output_image])
281
 
282
  gr.Markdown("""
283
  <center><h1>Summary</h1></center>
 
287
  We opted to use the [HAnd Gesture Recognition Image Dataset](https://github.com/hukenovs/hagrid) (HaGRID) and [MediaPipe's Hand Landmarker](https://developers.google.com/mediapipe/solutions/vision/hand_landmarker) to train a control net that could potentially be used independently or as an in-painting tool.
288
  <br>
289
  To preprocess the data there were three options we considered:
290
+ <be>
291
+ <ul>
292
+ <li>The first was to use Mediapipes built-in draw landmarks function. This was an obvious first choice however we noticed with low training steps that the model couldn't easily distinguish handedness and would often generate the wrong hand for the conditioning image.</li>
293
  <br>
294
+ <center>
 
295
  <table><tr>
296
  <td>
297
  <p align="center" style="padding: 10px">
 
310
  </tr></table>
311
  </center>
312
  <br>
313
+ <li>To counter this issue we changed the palm landmark colors with the intention to keep the color similar in order to learn that they provide similar information, but different to make the model know which hands were left or right.</li>
314
+ <br>
315
+ <center>
316
  <table><tr>
317
  <td>
318
  <p align="center" style="padding: 10px">
 
331
  </tr></table>
332
  </center>
333
  <br>
334
+ <li>The last option was to use <a href="https://ai.googleblog.com/2020/12/mediapipe-holistic-simultaneous-face.html">MediaPipe Holistic</a> to provide pose face and hand landmarks to the ControlNet. This method was promising in theory, however, the HaGRID dataset was not suitable for this method as the Holistic model performs poorly with partial body and obscurely cropped images.</li>
335
+ </ul>
336
  <br>
337
+ We anecdotally determined that when trained at lower steps the encoded hand model performed better than the standard MediaPipe model due to implied handedness. We theorize that with a larger dataset of more full-body hand and pose classifications, Holistic landmarks will provide the best images in the future however for the moment the hand-encoded model performs best. """)
338
 
339
 
340
  demo.launch()