Recent Text-to-Image (T2I) generation models such as Stable Diffusion and Imagen have made significant progress in generating high-resolution images based on text descriptions. However, many generated images still suffer from issues such as artifacts/implausibility, misalignment with text descriptions, and low aesthetic quality. In this paper, the authors propose enriching the feedback signal by collecting rich human feedback including marking implausible image regions, annotating misrepresented text terms, and providing fine-grained scores. They collected rich human feedback on 18,000 generated images and trained a multimodal transformer model (RAHF) to predict this feedback automatically. The predicted rich feedback is shown to improve image generation quality through data selection for finetuning and targeted inpainting, with improvements generalizing across different model families.

Text-to-image (T2I) generation models are rapidly becoming a key to content creation in various domains, including entertainment, art, design, and advertising, and are also being generalized to image editing, video generation, among many other applications. Despite significant recent advances, the outputs still usually suffer from issues such as artifacts/implausibility, misalignment with text descriptions, and low aesthetic quality. For example, in the Pick-a-Pic dataset, which mainly consists of images generated by Stable Diffusion model variants, many images contain distorted human/animal bodies (e.g., human hands with more than five fingers), distorted objects and implausibility issues such as a floating lamp.

Existing automatic evaluation metrics for generated images, including the well-known IS and FID, are computed over distributions of images and may not reflect nuances in individual images. Recent work has collected human preferences to train evaluation models, including ImageReward and Pick-a-Pic. However, these metrics still summarize the quality of one image into a single numeric score. Metrics such as CLIPScore exist for alignment evaluation, but these are expensive and complex models that still do not localize the regions of misalignment in the image. The authors argue that a much richer form of feedback is needed — one that is interpretable and attributable to specific image regions and text terms.

In this paper, the authors propose a dataset and a model of fine-grained multi-faceted evaluations that are interpretable and attributable (e.g., to regions with artifacts/implausibility or image-text misalignments), which provide a much richer understanding of image quality than single scalar scores. Their contributions include: (1) RichHF-18K, the first rich human feedback dataset on generated images, consisting of fine-grained scores, implausibility/misalignment image regions, and misalignment keywords on 18K Pick-a-Pic images; (2) a multimodal Transformer model (RAHF) to predict rich feedback, highly correlated with human annotations on the test set; (3) demonstrating usefulness through inpainting and finetuning; and (4) showing that improvements generalize to the Muse model, which differs from the models that generated the training images.

Text-to-image (T2I) generation models have evolved through several popular model architectures in the deep learning era. An early work is the Generative Adversarial Network (GAN), which trains a generator for image generation and a discriminator to distinguish between real and generated images. Another category develops from variational auto-encoders (VAEs), which optimize evidence lower bound (ELBO) for the likelihood of the image data. Most recently, Diffusion Models (DMs) have emerged as the state-of-the-art for image generation, trained to generate images progressively from random noise. Latent Diffusion Models are a further refinement that performs the diffusion process in a compact latent space for more efficiency.

There has been much recent work on evaluation of text-to-image models along many dimensions. Xu et al. collected human preference datasets and trained the ImageReward model. Kirstain et al. built the Pick-a-Pic dataset with over 500K examples of human preferences. Wu et al. collected large-scale human choices to train the Human Preference Score (HPS) classifier. Despite these valuable contributions, most existing works only use binary human ratings or preference ranking for construction of feedback/rewards, and lack the ability to provide detailed actionable feedback such as implausible regions of the image, misaligned regions, or misaligned keywords on the generated images.

For each generated image, the annotators are first asked to examine the image and read the text prompt used to generate it. Then, they mark points on the image to indicate the location of any implausibility/artifact or misalignment with respect to the text prompt. The annotators are told that each marked point has an "effective radius" (1/20 of the image height), which forms an imaginary disk centering at the marked point. In this way, a relatively small amount of points can cover the image regions with flaws. Lastly, annotators label the misaligned keywords and the four types of scores for plausibility, text-image alignment, aesthetic, and overall quality, respectively, on a 5-point Likert scale.

The authors designed a web UI to facilitate data collection with the following principles: (1) convenience for annotators to perform annotations ideally within a short time for an image-text pair, and (2) allowing annotators to perform all annotations on the same UI, so that the fine-grained scores are based on the annotated regions and keywords. The interface displays the image on the left and a panel on the right showing the text prompt. Annotators click the image to mark regions, then select misaligned keywords and scores on the right panel. All 27 annotators were trained with detailed annotation guidelines and calibrated before performing annotations, and the entire annotation process took approximately 3,000 rater-hours in total.

To improve the reliability of the collected human feedback, each image-text pair is annotated by three annotators, and the multiple annotations need to be consolidated. For the scores, the authors simply average the scores from the multiple annotators. For the misaligned keyword annotations, majority voting is performed to get the final sequence of aligned/misaligned indicators. For the point annotations, they are first converted to heatmaps, where each point is converted to a disk region on the heatmap, and then the average heatmap across annotators is computed. Regions with clear implausibility are likely to be annotated by all annotators and thus have a high value on the final average heatmap.

The authors select a subset of image-text pairs from the Pick-a-Pic dataset for data annotation. Although their method is general and applicable to any generated images, they choose the majority of the dataset to be photo-realistic images due to its importance and wider applications. To ensure balanced categories across the images, they utilized the PaLI visual question answering model to extract basic features and sample a diverse subset. The resulting RichHF-18K dataset consists of 16K training, 1K validation, and 1K test samples. Score distributions are approximately Gaussian, with about 25% of samples having perfect annotator agreement and approximately 85% having good agreement (maximum annotator score difference of at most 1 point on the 5-point Likert scale).

The architecture adopts a vision-language model based on ViT and T5X models, inspired by the Spotlight architecture but modifying both the model and pretraining datasets. The ViT takes the generated image as input and outputs image tokens as high-level representations. The text prompt tokens are embedded into dense vectors. The image tokens and embedded text tokens are concatenated and encoded by the Transformer self-attention encoder in T5X. This uses a self-attention module among the concatenated image tokens and text tokens, similar to PaLI, as the tasks require bidirectional information propagation — text information propagates to image tokens for misalignment heatmap prediction, while vision information propagates to text tokens for vision-aware text encoding.

On top of the encoded fused text and image tokens, three kinds of predictors are used. For heatmap prediction, the image tokens are reshaped into a feature map and sent through convolution layers, deconvolution layers, and sigmoid activation, outputting implausibility and misalignment heatmaps. For score prediction, the feature map is sent through convolution layers, linear layers, and sigmoid activation, resulting in scalars as fine-grained scores. For keyword misalignment sequence prediction, the original prompt is used as text input and a modified prompt with a special suffix ("_0") for each misaligned token is used as the prediction target for the T5X decoder. The authors explore two model variants: a multi-head version with seven separate prediction heads, and an augmented prompt version that uses a single head per prediction type with task-specific prompt augmentation.

The model is trained with a pixel-wise mean squared error (MSE) loss for heatmap prediction, and MSE loss for score prediction. For misalignment sequence prediction, the model is trained with teacher-forcing cross-entropy loss. The final loss function is the weighted combination of the heatmap MSE loss, score MSE loss, and the sequence teacher-forcing cross-entropy loss. To pretrain the model on more diverse images, the authors add the natural image captioning task on the WebLI dataset to the pretraining task mixture. The model uses ViT B16 (16x16 patch size) as the vision encoder and T5 base (12 layers) as the text encoder, trained with batch size 256 for 20K iterations using AdamW optimizer with learning rate 0.015.

The model is trained on the 16K RichHF-18K training samples, with hyperparameters tuned on the 1K validation set. For score prediction tasks, the authors report Pearson linear correlation coefficient (PLCC) and Spearman rank correlation coefficient (SRCC). For heatmap prediction, standard saliency metrics like NSS/KLD/AUC-Judd/SIM/CC are used for non-empty ground truth samples, while MSE is reported on all samples. For misaligned keyword sequence prediction, token-level precision, recall, and F1-score are adopted. Baselines include finetuned ResNet-50 models for scores and heatmaps, CLIP score for text-image alignment, and CLIP gradient map for misalignment heatmap prediction.

For score prediction, the two variants of the proposed model both significantly outperform ResNet-50 (or CLIP for text-image alignment score), with the augmented prompt version achieving PLCC of 0.693 for plausibility, 0.600 for aesthetics, 0.474 for text-image alignment, and 0.580 for overall score. For implausibility heatmaps, the multi-head version performs worse than ResNet-50, but the augmented prompt version outperforms ResNet-50 with MSE of 0.00920 and CC of 0.556. The main reason is that in the multi-head version, without augmenting the prediction task in the prompt, the same prompt is used for all seven prediction tasks, and hence the feature maps will be the same for all tasks, making it difficult to find a good tradeoff. After augmenting the prompt, the feature map can be adapted to each particular task with better results.

Additionally, misalignment heatmap prediction generally has worse results than artifact/implausibility heatmap prediction, possibly because misalignment regions are less well-defined, and the annotations may therefore be noisier. It is somewhat ambiguous how to label some misalignment cases such as absent objects on the image. The authors note this as a known limitation and improving the misalignment label quality is one of the future directions. Overall, the experiments demonstrate that the RAHF model, particularly the augmented prompt variant, can effectively predict rich human feedback annotations that are highly correlated with actual human judgments.

To ensure that gains from the RAHF model generalize across generative model families, the authors use Muse as their target model to improve, which is based on a masked transformer architecture and thus different from the Stable Diffusion model variants in the RichHF-18K dataset. First, they generate eight images for each of 12,564 prompts using the pre-trained Muse model. They predict RAHF scores for each image, and if the highest score exceeds a fixed threshold, it is selected as part of the finetuning dataset. The Muse model is then finetuned with this selected dataset. This approach could be viewed as a simplified version of Direct Preference Optimization. Human evaluation shows that the finetuned Muse with RAHF plausibility scores produces significantly fewer artifacts than the original Muse, with 51.83% of examples rated as significantly or slightly better.

The authors further demonstrate that predicted heatmaps and scores can be used to perform region inpainting to improve image quality. For each image, they first predict implausibility heatmaps, then create a mask by processing the heatmap using thresholding and dilating. Muse inpainting is applied within the masked region to generate new content matching the text prompt. Multiple images are generated, and the final image is chosen by the highest predicted plausibility score. The results show that more plausible images with fewer artifacts are generated after inpainting, demonstrating that RAHF generalizes well to images from a generative model very different from the ones whose images are used to train RAHF. The authors also demonstrate using the RAHF aesthetic score as Classifier Guidance to the Latent Diffusion model, showing that each fine-grained score can improve different aspects of generation.

In this work, the authors contributed RichHF-18K, the first rich human feedback dataset for image generation. They designed and trained a multimodal Transformer (RAHF) to predict rich human feedback, and demonstrated several ways to improve image generation with rich human feedback. While some results are quite exciting and promising, there are several limitations. First, the model performance on the misalignment heatmap is worse than on the implausibility heatmaps, possibly due to annotation noise. Second, it would be helpful to collect more data on generative models beyond Pick-a-Pic (Stable Diffusion). Moreover, while three promising leveraging approaches are presented, there is a myriad of other ways to utilize rich human feedback that can be explored, such as using predicted heatmaps as reward signals for reinforcement learning or as weighting maps during training.