This year, for the first time, we have enough activity in the federation to make scraping a sound search strategy.
We can know a site by fetching a summary via sitemap or all pages via export. Sitemap is widely available and usually fast. Export is unique to the node version but can be simulated with sitemap.
We discourage wide spread application of scrapers until we develop algorithms that avoid senseless duplication of server and network traffic.
Ruby Sitemap Scrape with search application.
Javascript Sitemap Scrape with animation.
Ruby Export Scrape and subsequent tallies.
Distributed Search looking for more.
Find More related to any page many ways.
# Issues
Search is great and hugely powerful. The danger is that it is too powerful. It delivers what is needed without the requirement to explore, make relationships and evolve content slowly through a social process of Writing with Strangers.
With the addition to the Fedwiki IPFS Plugin, we now have the ability to create a Semi-Private p2p wiki, in which authors are free to write together with other like minded people without the Fear of Early Exposure. A comprehensive search functionality, whle needed risks destroying this way of writing.
As the Federation grows, as new members and more content becomes available, the value to others with commercial, or simple ill-intention becomes greater.
Conclusion: yes, Web search engines can index wikis. Google Loves Wiki.