At its core, building a Sora-like system requires an architecture that unifies local image/video detail generation (via diffusion) with long-range temporal and structural reasoning (via transformer / attention modules) over spatio-temporal patches. The model must operate in a compressed latent space to reduce compute and memory costs, and then decode to full video frames. According to public writeups, Sora uses a “diffusion transformer” model (sometimes called DiT) that treats patches in space and time as tokens and performs denoising under attention over those tokens.
The architecture pipeline broadly looks like this: first, the input text is enriched (recaptioned) to add descriptive detail; then that embedding conditions the diffusion process. The video latent space is partitioned into patches (x, y, t) and the diffusion transformer works on these tokens, applying attention across time and space so that objects remain consistent, motion trajectories make sense, and scene layout is stable. After the diffusion steps, a decoder (or video decompressor) transforms the latent video back to pixel frames. Because video generation is heavy, careful choices of patch granularity, memory management, and attention sparsity are necessary to make it tractable. In more efficient variants of diffusion transformer models, methods like sparse 3D attention and step distillation are used to reduce computational cost. For example, work like Efficient-vDiT shows how pruning redundant 3D attention patterns and shortening the sampling process can yield large speedups.
In short, the needed architecture blends diffusion (for detail) with transformers (for structure) over spatio-temporal tokenization, all within a compressed latent domain. To make this work in practice, you also need modules for prompt conditioning, efficient attention, memory optimization, and a decoder to recover video frames—all while balancing fidelity, coherence, and computational cost.