In practice it is certainly true that any word that gathers strength enough to become buzzy gets coopted by dominant forces that prefer not to truly engage with any concept that challenges their dominance. This is why terminology keeps shifting, new words replace the old ones deemed less worthy by the taint of dominant-ideology faux acceptance.
Diversity is infinite. Diversity is reality. Diversity is material and temporal; change is diversity occurring in time. Diversity envelops all things, even those things that think they can own it.
Maybe it depends on who you think we is. When we say we need diverse books, the we is not singular in any way. If you think that we is white people, then diversity is not a useful term. But ultimately the we of “We need diverse books” is Diversity itself. Diversity needs diverse books; further, it demands them.