검색 상세

Personalizing Diffusion Inpainting Model with Text-Free Finetuning

  • 김범조 (BEOMJO KIM, 아주대학교 일반대학원)

초록/요약

This thesis introduces a novel approach to subject-driven image generation, advancing the field by overcoming the limitations of traditional text-to-image diffusion models. Our method employs a model that generates images from reference images without the need for language-based prompts. By integrating our proposed module named as visual detail preserving module, the model captures intricate visual details and textures of subjects, addressing the common challenge of overfitting associated with a limited number of training samples. We further refine the model's performance through a modified classifier-free guidance technique and feature concatenation, enabling the generation of images where subjects are naturally positioned and harmonized within diverse scene contexts. Quantitative assessments using CLIP and DINO scores, complemented by a user study, demonstrate our model's superiority in fidelity, editability, and overall quality of generated images. Our contributions not only show the potential of leveraging pre-trained models and visual patch embeddings in subject-driven editing but also highlight the balance between diversity and fidelity in image generation tasks. Keywords: Diffusion Model, Image Generation, Image Inpainting, Subject-Driven Generation, Image Manipulation

more

목차

1. Introduction 1
2. Related Works 5
2.1 Diffusion Model 5
2.2 Subject-Driven Generation 7
2.3 Controlling Pre-trained Diffusion Models . 8
2.4 Diffusion Models Inference with Guidance . 9
3. Method 12
3.1 Preliminaries 12
3.2 Training Phase 14
3.2.1 Feature Extraction . 15
3.2.2 Feature Injection 16
3.3 Inference Phase . 17
4. Experiments 20
4.1 Experiments Details 20
4.2 Comparable Results. 22
4.2.1 Qualitative Result 23
4.2.2 Quantitative Result 24
4.3 Ablation Study 26
5. Conclusion 29
Discussions and Future works 29
Reference 31

more
반출 Meta View 목록