Can search tame the wild web? Can open access help?Which is growing faster, the size of the web or the power of search engines? No one knows, but it matters for all of us. If the power of search is growing faster, then it will eventually overtake the web's wildness and tame it --all of it. If the web is growing faster, then it will forever outpace our attempts to map, navigate, explore, and understand it.
SPARC Open Access Newsletter, issue #92
December 2, 2005
by Peter Suber
The problem reminds me of the critical density of the universe. If there is enough matter in the universe, then gravitational attraction will eventually slow down and reverse cosmic expansion. The universe will end in a big crunch. If there isn't enough matter in the universe, the expansion will continue unimpeded indefinitely. The universe will end in heat death. When I was a student, I don't think we knew which outcome was more likely. But the recent discovery of accelerated expansion, perhaps due to dark energy, means that expansion is outpacing gravity.
We may not have a preference between the big crunch and heat death, especially if they're both billions of years in the future. But we should definitely root for the power of search to overtake the wild and rapid expansion of the web. If it does, then as readers we'll be able to find what we want and as authors we'll have a chance of being found by readers. But if search can't keep up with expansion, then as John Alan Paulos put it, the web will be the world's largest library, but its books will be scattered all over the floor.
If the power of search is above the critical threshold, then users will eventually have adequate maps of this wilderness. Search will not only overtake and virtually slow the expansion, but also reverse it and lead to crunch. Once every page is indexed, and we can keep pace with future growth, then we can add new and more useful layers of organization to the index, such as semantic connections that go beyond coded links, "crunching" the web into ever more useful forms.
Unfortunately it's difficult or impossible to make a direct comparison between the rate of web growth and the rate of search improvement. One reason is that web growth and search power are not forces of nature (limiting my analogy). Either or both may suddenly speed up or slow down. We aren't surprised when they fluctuate because we know they are functions of a thousand variables that we do not yet fully understand.
There are many reasons why search may not keep pace with the expanding web. One is limited scope. What percentage of the web does a search engine index? A critical part of search scope is the dark web, a.k.a. the deep or invisible web stored in databases and closed to search engine crawlers. By one estimate the deep web is 500 times larger than the surface web.
Another is the adequacy of the relevance algorithm. How good is the search engine at guessing what you want and giving it a favored position in the search returns? Another is search spam. How good is the search engine at resisting attempts to manipulate its index and ranking algorithm? Another is lack of personalization. Even if the search index is complete and the relevance algorithm top-notch, different users may have different needs when searching on the same search term. Some will find what they want at the top of the relevance-sorted list but others won't, and it doesn't help them to know that what they seek is listed somewhere among the millions of other sites further down the list. (If you insist that search spam and lack of personalization are problems within the relevance algorithm, not separate problems beyond it, then I'll agree.)
The dark web is like dark energy, which accelerates expansion, not dark matter, which slows it down. By resisting search, the dark web helps the cause of entropy and hurts the cause of closure and organization.
Some of the literature in the dark web can still be searched by speciality search engines like Copernic, Deep Query Manager, iBoogie, and ProFusion. Moreover, even when dark web content is invisible to outside tools, it is usually hosted by databases that offer their own search functions. But even together these tools don't take us very far. Most of the dark web is still an unmapped region where content expands unnoticed by web-wide search engines.
Price barriers seclude content just as databases do and create a similar kind of web darkness. Some publishers of non-OA content invite major search engines like Google and Yahoo to index them, and I applaud that. And some search engines, like Scirus, already cover segments of the non-OA literature because they are created by the publishers. But most toll-access literature is still invisible to the general search engines.
In this sense, open access contributes to the mapping, taming, organization, and intelligibility of the web, while toll access resists the forces of closure.
Search is the gravity of the online universe that holds everything together, if anything does. As such, open access is one of its greatest allies. One thing we can do to defeat web entropy and help the cause of organization and discovery is to provide open access to a continually growing proportion of research literature. The harder this literature is to index --and price barriers increase the difficulty-- the harder it is for search to keep pace with the relentless expansion.
* Postscript. I haven't seen other discussions of the analogy between cosmic expansion and web expansion, and the power of gravity and the power of search. But I can't believe I'm the first to notice it. If anyone can point out earlier discussions, I'd be glad to credit the pioneers.
----------
Read this issue online
http://dash.harvard.edu/bitstream/handle/1/3942942/suber_news92.htmlSOAN is published and sponsored by the Scholarly Publishing and Academic Resources Coalition (SPARC).
http://www.arl.org/sparc/Additional support is provided by Data Conversion Laboratory (DCL), experts in converting research documents to XML.
http://www.dclab.com/public_access.asp
==========
This is the SPARC Open Access Newsletter (ISSN 1546-7821), written by Peter Suber and published by SPARC. The views I express in this newsletter are my own and do not necessarily reflect those of SPARC or other sponsors.
To unsubscribe, send any message (from the subscribed address) to <SPARC-OANews-off@arl.org>.
Please feel free to forward any issue of the newsletter to interested colleagues. If you are reading a forwarded copy, see the instructions for subscribing at either of the next two sites below.
SPARC home page for the Open Access Newsletter and Open Access Forum
http://www.arl.org/sparc/publications/soan
Peter Suber's page of related information, including the newsletter editorial position
http://www.earlham.edu/~peters/fos/index.htm
Newsletter, archived back issues
http://www.earlham.edu/~peters/fos/newsletter/archive.htm
Forum, archived postings
https://mx2.arl.org/Lists/SOA-Forum/List.html
Conferences Related to the Open Access Movement
http://www.earlham.edu/~peters/fos/conf.htm
Timeline of the Open Access Movement
http://www.earlham.edu/~peters/fos/timeline.htm
Open Access Overview
http://www.earlham.edu/~peters/fos/overview.htm
Open Access News blog
http://www.earlham.edu/~peters/fos/fosblog.html
Peter Suber
http://www.earlham.edu/~peters
peter.suber@earlham.edu
SOAN is licensed under a Creative Commons Attribution 3.0 United States License.
http://creativecommons.org/licenses/by/3.0/us/
Return to the Newsletter archive