Google Says Using A CDN & Want One Robots.txt File, Redirect Yours To The CDN

Date:


Google Crossing

Do you use a CDN for some or all of your website and you want to manage just one robots.txt file, instead of both the CDN’s robots.txt file and your main site’s robots.txt file? Gary Illyes from Google said that all you need to do is redirect your main robots.txt to the CDN and control it on the CDN.

Gary wrote on LinkedIn:

You have a CDN and you have your main site. You have two robots.txt files, one at https∶//cdn․example․com/robots.txt and one at https∶//www․example․com/robots.txt. You could have just one central robots.txt with all the rules, say on your CDN, which might help you with keeping track of all rules you need to manage. All you have to do is to redirect https∶//www․example․com/robots.txt to https∶//cdn․example․com/robots.txt and crawlers that comply to RFC9309 will just use the redirect target as the robotstxt file of https∶//www․example․com/. Weird or what.

Now, this is not an SEO ranking tip. It is just a way to keep everything under one robots.txt and not have to manage multiple. This is not going to help you rank better or anything like that.

I am now waiting to see ads for SEO services to move your robots.txt file to a CDN. Again, this does not give you any SEO benefit or ranking benefit, this is just to make it easier for you when you want to maintain one file. Of course, if you ever leave that CDN, then you need to deal with that, but it isn’t much different than moving to a new host.

Forum discussion at LinkedIn.



Source link

Share post:

[tds_leads title_text="Subscribe" input_placeholder="Email address" btn_horiz_align="content-horiz-center" pp_checkbox="yes" pp_msg="SSd2ZSUyMHJlYWQlMjBhbmQlMjBhY2NlcHQlMjB0aGUlMjAlM0NhJTIwaHJlZiUzRCUyMiUyMyUyMiUzRVByaXZhY3klMjBQb2xpY3klM0MlMkZhJTNFLg==" f_title_font_family="653" f_title_font_size="eyJhbGwiOiIyNCIsInBvcnRyYWl0IjoiMjAiLCJsYW5kc2NhcGUiOiIyMiJ9" f_title_font_line_height="1" f_title_font_weight="700" f_title_font_spacing="-1" msg_composer="success" display="column" gap="10" input_padd="eyJhbGwiOiIxNXB4IDEwcHgiLCJsYW5kc2NhcGUiOiIxMnB4IDhweCIsInBvcnRyYWl0IjoiMTBweCA2cHgifQ==" input_border="1" btn_text="I want in" btn_tdicon="tdc-font-tdmp tdc-font-tdmp-arrow-right" btn_icon_size="eyJhbGwiOiIxOSIsImxhbmRzY2FwZSI6IjE3IiwicG9ydHJhaXQiOiIxNSJ9" btn_icon_space="eyJhbGwiOiI1IiwicG9ydHJhaXQiOiIzIn0=" btn_radius="3" input_radius="3" f_msg_font_family="653" f_msg_font_size="eyJhbGwiOiIxMyIsInBvcnRyYWl0IjoiMTIifQ==" f_msg_font_weight="600" f_msg_font_line_height="1.4" f_input_font_family="653" f_input_font_size="eyJhbGwiOiIxNCIsImxhbmRzY2FwZSI6IjEzIiwicG9ydHJhaXQiOiIxMiJ9" f_input_font_line_height="1.2" f_btn_font_family="653" f_input_font_weight="500" f_btn_font_size="eyJhbGwiOiIxMyIsImxhbmRzY2FwZSI6IjEyIiwicG9ydHJhaXQiOiIxMSJ9" f_btn_font_line_height="1.2" f_btn_font_weight="700" f_pp_font_family="653" f_pp_font_size="eyJhbGwiOiIxMyIsImxhbmRzY2FwZSI6IjEyIiwicG9ydHJhaXQiOiIxMSJ9" f_pp_font_line_height="1.2" pp_check_color="#000000" pp_check_color_a="#ec3535" pp_check_color_a_h="#c11f1f" f_btn_font_transform="uppercase" tdc_css="eyJhbGwiOnsibWFyZ2luLWJvdHRvbSI6IjQwIiwiZGlzcGxheSI6IiJ9LCJsYW5kc2NhcGUiOnsibWFyZ2luLWJvdHRvbSI6IjM1IiwiZGlzcGxheSI6IiJ9LCJsYW5kc2NhcGVfbWF4X3dpZHRoIjoxMTQwLCJsYW5kc2NhcGVfbWluX3dpZHRoIjoxMDE5LCJwb3J0cmFpdCI6eyJtYXJnaW4tYm90dG9tIjoiMzAiLCJkaXNwbGF5IjoiIn0sInBvcnRyYWl0X21heF93aWR0aCI6MTAxOCwicG9ydHJhaXRfbWluX3dpZHRoIjo3Njh9" msg_succ_radius="2" btn_bg="#ec3535" btn_bg_h="#c11f1f" title_space="eyJwb3J0cmFpdCI6IjEyIiwibGFuZHNjYXBlIjoiMTQiLCJhbGwiOiIxOCJ9" msg_space="eyJsYW5kc2NhcGUiOiIwIDAgMTJweCJ9" btn_padd="eyJsYW5kc2NhcGUiOiIxMiIsInBvcnRyYWl0IjoiMTBweCJ9" msg_padd="eyJwb3J0cmFpdCI6IjZweCAxMHB4In0="]
spot_imgspot_img

Popular

More like this
Related

Why This Expert Investor Owns Only ONE Rental Property

Mike Baum owns just one rental property, but...

The 12 Days of Christmas If Written by Dogs – Dogster

Welcome to the Wednesday Dogs of Dogster! Every...

2022 Gardening Know How Sponsorship Recipients

At Gardening Know How we support school and...

WATCH: 24 must-see moments of 2024 – Does your favourite video feature?

WATCH: 24 must-see moments of 2024 - Does...