I've also been asking myself this question since I read Human interface. Although Jef Raskin presented his ZoomWorld as a complete idea, the real expert in ZUI is Ben Bederson. He has been building and studying zoom interfaces for decades. He also has some of his own answers to his question. He summed up, ‘Manual organization of information in a zoom environment rarely makes sense’. He wrote a full article on the question, "The promise of user interfaces with zoom." From the section "Why ZUIs are challenging" (pp. 5–6):
As summarized in Table 1, the potential benefits of ZUI are sometimes mirages. ZUIs are generally attractive (although they make some people feel physically ill) and visually rich. But the promise of simplicity falls short.
While human visual perception makes it easy to see where you are navigating, the reality is that it places a great load on short-term memory to remember where it just was in space and where things are. And the requirement of human memory to know how space is organized means that the ZUIs do not expand very well. ZUIs are often motivated by the physical world and the way people like to put papers on their desk. But nobody wants everyone of his papers on his desk. It is much more common to have only a relatively small number of documents that you are really working with.
The visual descriptions that ZUIs offer for free when moving away may seem like a solution to the burden on human memory, but in practice they do not because visual descriptions of any complexity require significant scanning and a visual search to find something. If there are only a small number of objects, the visual search task is not difficult, but, of course, for a small number of objects, you do not need a ZUI to solve your organizational problems.
Finally, the visual richness of the ZUI is a double-edged sword. It requires skill to design a complex space with documents of arbitrary size, aspect ratio and color that people can understand and scan. Also, people are not as good at scanning 2D designs as 1D designs, unless the design is highly structured. But highly structured 2D designs do not work well for visual objects of arbitrary aspect ratios. Designers are obliged to leave a large amount of unused space, reduce large objects so that large objects cannot be read or trimmed, thus losing much of their distinction.
I would like to add some points of my own. First, Jef Raskin's zoom interface makes more sense in a document / object-centric context. Application-centric environments are conceptually at odds with such interfaces. The Raskin ZUI is good for organizing large amounts of information so that applications become obsolete; but the current computer landscape is about "applications" (following Apple's example, from the original Macintosh to the iPhone). Even the iOS home screen, a kind of smaller zoom interface (especially after adding nested folders), is restricted to the organization of application icons, not real content.
Second, paradigm changes tend to take a long time. Concepts such as graphic interfaces and direct manipulation were invented in the 1960s, but it took until the mid-1980s to make these paradigms available to the general public. Internet and hyperlinked documents took even longer. The multi-touch screens date back to the 80s, but the iPhone was the first affordable implementation available in general. Zoomed interfaces often require more resources; by the time they were practical, traditional GUIs were quite established.
I believe that a "killer application" is necessary to facilitate this paradigm shift focused on the object-centered application. In my opinion, the best way to answer why ZUIs are not more popular is to deny the premise: build one! (That is my own plan).