Abstract: Generating high-fidelity images in specialized domains like computational pathology or remote sensing is often hampered by a lack of labeled data. To address this challenge, we introduce two novel approaches. First, we present PathLDM, a text-conditioned Latent Diffusion Model that leverages the rich information in histopathology reports to guide the generation of high-quality histopathology images. Through strategic conditioning and necessary architectural enhancements, PathLDM achieves a SoTA FID score of 7.64 for text-to-image generation on the TCGA-BRCA dataset, significantly outperforming the closest text-conditioned competitor with FID 30.1. While histopathology reports offer valuable clinical insight, they lack granular patch-level guidance. To overcome this limitation, we develop a second approach that conditions diffusion models on embeddings derived from self-supervised learning (SSL) encoders. Our diffusion models successfully project these features back to high-quality histopathology and remote sensing images. In addition, we construct larger images by assembling spatially consistent patches inferred from SSL embeddings, preserving long-range dependencies. The SSL embeddings used to generate a large image can either be extracted from a reference image, or sampled from an auxiliary model conditioned on any related modality (e.g. class labels, text, genomic data). As proof of concept, we introduce the text-to-large image synthesis paradigm where we successfully synthesize large pathology and satellite images out of text descriptions.