I know just how you feel. Years ago I ripped out my N scale layout which had over 11 scale miles of main line track mainly due to the poor quality of the locomotives that were available at the time. I was able to use some of the bench work from the N scale layout but was hand laying my track in HO scale which at times was painfully slow, but you do have an extremely good point about not rushing things. Taking your time to lay track smoothly without having any problems is so important.
Having a plan is also extremely important. I never drew out a track plan, but I knew what kind of a layout I wanted. I knew that I wanted to model the transition era, growing up when steam was still running and I also liked the first generation diesels. I knew that I wanted to have a switching layout because I really didn't have the room to run long trains (although I can do that it I want).
Progress was slow. had 8 feet of track down and using alligator clips I ran a locomotive and a few cars over that section of track just to see a train run. The 8 feet grew into 16 feet and then 25 feet and on and on. I did choose to freelance and chose the area right where I live as I knew what industries were in the area, or what could have existed if a railroad were there to serve the industries. I took my time to run trains through all of the turnouts to the industries making sure that the track work was smooth and that there wouldn't be any derailment problems.
I had the structures, or at least the footprint of a building and set them out on the layout and ran track to the industries trying to make them look somewhat realistic as to how they would appear in a town. I did screw up in one town ending up with cattle loading pens set right behind a diner. Not a good idea. Had to tear out some tracks and move the stock pens out of town so people eating would not have to smell the cow poop.
In the last year, after over 30 years of building on the layout, my last town, yard and engine servicing facility was finally in place. Now it's time to enjoy operating the layout, just as it was designed. Can't count how many times I have gone down to the train room to work on a project and just ended up running trains.
Yes, it did take a long time for me to get where I am, but I have other hobbies and I also don't have any hobby shops where I live. There was a period of probably 8 to 10 years where nothing was done to the layout because it was close to impossible to get what I needed to move forward with the layout. Internet retailers saved my butt and now I I ca pay attention to adding details and updating older parts of the layout.
You are doing the right thing by taking your time.
