Someone said they would never hire fat people, explaining that overweight people could not be trusted because if they lost the most precious thing in life - their body, their health - how could you trust them not to ruin the business they were hired to do?
While I don't endorse this opinion, I have to admit that it has been on my mind for about a year now.
I mean, I've been thinking about my body and my health.
“Turns out, the Robots.txt file was preventing search engines from crawling the archives folder, which then meant that Google couldn’t then read the robots meta tags and know to noindex the pages. Damn.”
There is - obvs the robots. txt but if urls are indexed you usually want them out of the index so need to let Google read the pages noindex - then turn off using robots.txt Alternatively you can remove those urls from Google using Search Console. 😀
Additionally Google will be following links from other sites to your old pages. Even if you get them out of the index it is still going to be looking for them because of the offsite links. And your old posts are going to have a lot of links!
@ilovechoclates_
Yep. I have urls from hundreds of years ago still in the index.
If you realy want that page to be noindexed then change the url - redirect the old one to something else appropriate and of course keep the noindex.
@ilovechoclates_
You could use Screaming Frog to find all the links in your site and then you have your changes list - cannot be that onerous! Otherwise you could give the remove URL tool in GSC a go - its only temporary, 6 months i have heard but that probably depends. Never measured that myself.