@lmika My dad's always talking about how people should do that! He also talks about how it should be possible, with enough source photos, to construct 3D models of those kinds of old buildings (he especially wants it to happen for all the heritage architecture that gets bulldozed to build flats). But certainly for those places that are important to us personally, it also makes good sense.
@jayeless Microsoft Photosynth was doing this 2006. Not sure what the status of that is these days.