Project Overview
Pixel-art sprites are a core asset type in game development, but producing them is manual and slow, and the datasets available for training generative models are small and style-specific. Models trained on a single style tend to produce noisy outputs with limited variety.
We studied whether combining two distinct sprite styles, detailed character-focused Pokémon sprites and simpler environment-focused Tiny Hero sprites, could reduce the data-scarcity problem while preserving what makes each style recognizable. Sprites were resized to a common resolution and augmented with flips, rotations, and color shifts.
generative model families compared: diffusion, beta-VAE, and GM-VAE
best generation score (our diffusion model), beating every VAE baseline on both styles
generated samples per style used for FID evaluation, plus 1,500 test samples per style for reconstruction
My Role and Contributions
This was an equal-contribution group study of four (with Faraz Khadivpour, Mehrshad Tavana, and Tahsin Mostafa) for the Machine Learning for Procedural Content Generation course at the University of Alberta.
I implemented and fine-tuned the diffusion model, the study's winning architecture, together with Mehrshad, and wrote the results and conclusions. The diffusion model used a residual U-Net with sinusoidal time embeddings (embedding dimension 32), layer widths of 32-64-96-128 with block depth 2, and skip connections to preserve detail through the denoising process.
Supervisor
Matthew Guzdial
Associate Professor, University of Alberta
Team
Faraz Khadivpour, Mehrshad Tavana,
Tahsin Mostafa, Muhammad Talha
Research Question
Can a multi-style training corpus overcome the data scarcity that limits single-style generative models, and which model family best balances generation quality, reconstruction fidelity, and controllable style behavior for pixel art?
Why it is hard
Pixel art has tiny resolutions where every pixel matters, so models that blur or average detail fail visibly. And when two styles share one model, the model must generalize across styles without collapsing them into a muddled average.
Study Design
Dataset Construction
Combined Pokémon and Tiny Hero sprites at a common resolution with flip, rotation, and color augmentation.
Model Training
Trained a residual U-Net diffusion model, beta-VAEs across four beta values (0.01 to 10) with a PixelSight layer, and a GM-VAE with style-clustering labels.
Evaluation
Four criteria: FID on 1,000 generated samples per style, SSIM and MSE reconstruction on 1,500 test samples per style, t-SNE latent visualization, and latent interpolation.
Analysis
Compared model families per style and per criterion, separating latent-space quality from generation quality.
Results
The diffusion model won generation on both styles: FID 32.70 on Pokémon and 67.19 on Tiny Hero, versus 47.16 and 69.85 for the best beta-VAE settings and 185.56 and 233.56 for GM-VAE. It also delivered the best reconstruction of the complex Pokémon style (SSIM 0.8642, MSE 0.0129).
Pokémon-style outputs
Crisp detail from the diffusion model (center) next to a washed-out VAE output (right).
Tiny Hero outputs
The same pattern on the simpler style: sharp edges survive diffusion, blur creeps into weaker models.
The VAEs told a complementary story: a small beta value (0.01) reconstructed the simpler Tiny Hero style best (SSIM 0.9403), and GM-VAE produced the cleanest style clusters in t-SNE space even though its generations were weakest.
Mixed latent space
Beta-VAE at β = 1: the two styles blend into one cloud.
Clean style clusters
GM-VAE separates the styles beautifully, yet generates the weakest sprites.
Key Insight
Separation in latent representation does not imply generation quality. GM-VAE clustered the two styles beautifully and generated poorly; the diffusion model, which has no comparable latent space, generated best. Model choice should follow the task, not the elegance of the latent space.
Key Contributions
Winning Diffusion Model
Implemented and fine-tuned the residual U-Net diffusion model that outperformed all VAE baselines on FID for both sprite styles.
Multi-Style Training Corpus
Showed that combining distinct sprite styles mitigates data scarcity while preserving per-style characteristics.
Multi-Criteria Evaluation
Evaluated generation, reconstruction, and latent behavior separately, revealing that they do not rank models the same way.
Team Study
Equal-contribution study with Faraz Khadivpour, Mehrshad Tavana, and Tahsin Mostafa, each owning a model family.
Skills, Tools & Frameworks
Code
All three model notebooks (diffusion, beta-VAE, GM-VAE) with the evaluation results are available on GitHub.
GitHub Repository