I think your idea is a good idea. You could keep the view distance low on the server to reduce the number of real chunks being generated. Clients can render whatever they want.
I think one issue would be when you approach a player structure from the outside, the client might see an unmodified world first and then the player structure would pop in and overwrite the client-side terrain. Its not a technical issue, but more a player experience concern.