Do visual tasks have a relationship to each other? The authors propose a fully computational approach for modeling the structure of the space of visual tasks. This is done via finding transfer learning dependencies across 26 tasks, computing a taxonomy (task structure) using transfer learning as the computational probe. They show that the structure in the space of visual tasks is real, discoverable, and useful: e.g., the total number of labeled datapoints needed for solving a set of 10 tasks can be reduced by roughly 2/3 while maintaining performance, by exploiting the relationships between tasks through systematic transfer.

Computer vision has traditionally solved tasks in isolation, treating each problem independently. Yet it seems natural that tasks should be related — a system that can detect edges might more easily learn to estimate surface normals, and a model that understands depth might transfer well to scene layout estimation. The authors argue that "a model aware of the relationships among tasks demands less supervision, uses less computation, and behaves in more predictable ways." However, the relationships between tasks are largely unknown and are currently established ad hoc or based on intuition.

This paper proposes a fully computational approach to map out the task structure. The approach involves training task-specific networks for each of 26 visual tasks, then systematically probing transfer learning performance from every source task to every target task. The resulting affinity matrix is normalized and solved as a combinatorial optimization problem to produce a taxonomy — a principled mapping of which tasks should transfer to which. The dataset consists of 4 million images from indoor scenes, annotated for all 26 tasks.

The work relates to several research areas. Multi-task learning jointly trains models on multiple tasks but does not explicitly model the relations among tasks or extract a meta-structure. Transfer learning and domain adaptation focus on transferring knowledge between tasks or domains, but usually between a single source-target pair. Self-supervised learning discovers representations from unlabeled data, but the choice of pretext task is typically based on intuition rather than a principled understanding of task relationships. Unlike all prior approaches, this work explicitly models the full structure of the task space to guide transfer decisions.

The first step trains independent encoder-decoder networks for each of the 26 tasks. All networks share a common encoder architecture to ensure that representations are comparable across tasks. The encoder is based on a ResNet-like architecture producing a fixed-size representation, while decoders are task-specific. The 26 tasks span four categories: 2D tasks (e.g., edge detection, texture), 2.5D tasks (e.g., depth, surface normals), 3D tasks (e.g., scene layout, vanishing points), and semantic tasks (e.g., object classification, semantic segmentation).

In the second step, small readout functions are trained to map from the frozen encoder representation of a source task to the target task. These transfer networks are deliberately kept shallow (1-2 layers) to measure whether information is "easily extractable" rather than whether it can be computed de novo. This is done for all source-target pairs, yielding a 26x26 transfer affinity matrix. Additionally, higher-order transfers are tested, where representations from multiple source tasks are combined to predict a single target.

Different tasks use different loss functions with incomparable scales, making direct comparison of transfer qualities impossible. To address this, the authors employ the Analytic Hierarchy Process (AHP), a decision-making framework that uses pairwise tournament matrices to normalize affinities. For each target task, source tasks are compared pairwise based on their transfer performance, producing ordinal rankings that are scale-invariant.

The final step formulates taxonomy computation as a Binary Integer Programming (BIP) problem. Given a supervision budget (how many tasks can be trained from scratch), the optimization selects a subgraph of the task transfer graph that maximizes total transfer performance. The BIP constraint ensures each target task receives transfer from at most a specified number of source tasks. The resulting taxonomy is a directed hypergraph where edges represent feasible transfers, weighted by expected performance gains.

The taxonomy is evaluated across multiple dimensions. First, task-specific network quality is validated: win rates versus random representations ranged from 60.2% to 100% across tasks, confirming that encoders learn meaningful features. Second, the computed taxonomy outperformed all other connectivities by a large margin — including random transfer policies, expert intuition-based transfers, and single-source transfers. Third, the approach demonstrates strong generalization: cross-dataset validation on Places and ImageNet achieved Spearman correlations of 0.857 and 0.823 respectively. Perhaps most strikingly, solving 10 tasks requires only ~37% of the labeled data needed for independent training.

Several surprising findings emerge from the taxonomy. Surface normals transfer to depth estimation better than vice versa, contradicting the common intuition that depth is more "fundamental." 2D texture and edge features provide surprisingly strong transfer to 3D tasks. Higher-order transfers — combining multiple sources — often outperform any single-source transfer, suggesting that tasks provide complementary information that can be leveraged jointly. The taxonomy also reveals clear cluster structures: semantic tasks form one cluster, geometric tasks another, with 2.5D tasks serving as bridges.

The paper demonstrates that the space of visual tasks possesses a discoverable, exploitable structure. The proposed computational approach — combining task-specific networks, transfer probing, normalization, and combinatorial optimization — yields a taxonomy that significantly reduces the supervision needed for solving multiple tasks simultaneously. The authors note several limitations: the findings are dependent on the specific model architecture and dataset, and transitive transfer chaining was found to be ineffective. The taxonomy, dataset, models, and an interactive solver are made publicly available at taskonomy.vision.