What is the smallest practical size for a layout in HO scale?

For a switching layout the minimum size can be as small as 4 x 1 feet. This can include a small 1 track staging track, and some sidings and maybe a passenger depot.

For a continuous loop railroad with at least 1 turnout, and standard minimum curves of 18 inch radius, the size of the layout needs to be at least 3 feet 6 inches x 4 feet. But this size layout does not give much scope for operation. To have a model railroad that is based on a continuous loop, and is interesting to operate, the minimum size would be closer to 6x4 feet. If you don't have much space, you might want to consider the potential for a shwitching layout, as the narrower width of these types of layouts means they could fit along the wall of almost any room, whether that room is being used for other purposes or not.