Here we learn the real application for prompt based image synthesis:
Some adventure game fans at Google wondered what would happen if you used the output from the classic text adventure game "Zork" as the input to Imagen, Google's text-to-image diffusion model. To support the results we wanted we made some additions to Zork itself, which was a fascinating journey through an arcane language and a lost but now re-engineered toolchain.
Nishat via matrix 
Hello
Ward
: Thanks for the Zork post. I'm quite interested in these kind of projects and data visualization projects. My masters thesis was based upon that and published as a research paper
Image Synthesis with Semantic Diffusion Guidance github.io
Controllable image synthesis models allow creation of diverse images based on text instructions or guidance from an example image. Recently, denoising diffusion probabilistic models have been shown to generate more realistic imagery than prior methods, and have been successfully demonstrated in unconditional and class-conditional settings.
We explore fine-grained, continuous control of this model class, and introduce a novel unified framework for semantic diffusion guidance, which allows either language or image guidance, or both. Guidance is injected into a pretrained unconditional diffusion model using the gradient of image-text or image matching scores. We explore CLIP-based textual guidance as well as both content and style-based image guidance in a unified form. Our text-guided synthesis approach can be applied to datasets without associated text annotations. We conduct experiments on FFHQ and LSUN datasets, and show results on fine-grained text-guided image synthesis, synthesis of images related to a style or content example image, and examples with both textual and image guidance.
~
LIU, Xihui, PARK, Dong Huk, AZADI, Samaneh, ZHANG, Gong, CHOPIKYAN, Arman, HU, Yuxiao, SHI, Humphrey, ROHRBACH, Anna and DARRELL, Trevor, 2023. More Control for Free! Image Synthesis With Semantic Diffusion Guidance. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. Online. 2023. p. 289–299. [Accessed 19 January 2023]. Available from: https://openaccess.thecvf.com/content/WACV2023/html/Liu_More_Control_for_Free_Image_Synthesis_With_Semantic_Diffusion_Guidance_WACV_2023_paper.html