I have a service that catalogs every file on every medium present, here. It is smart enough to dig into various archive formats (zip, tar, ace, etc.) and other containers (vhd, vmdk) to find files that they can contain. And, it does so recursively (so, a tgz located in a vmdk that happens to contain a vhd which holds a cpio archive... can be explored).
A colleague has asked me to extend this for his use *beyond* his local network segment.
E.g., he'd like to be able to point it at various Internet resources and have those contents catalogued, as well.
As I'm not keen on undertaking a major redesign, I'm just looking to wrap those resources in a FUSE and just add hooks to mount the resources in question thusly.
But, there are lots of ways that resources are exposed to The Internet (WebDAV, FTP sites, HTTP-wrapped folders, etc.).
Any observations of what seem to be the most common? [I'm not interested in cloud provided services as they can change their interfaces on a whim]