For decades, the owners of 323 West 80th Street took in wayward actors, aspiring musicians, and a rotating cast of models.