|
Ignoring server robots settings is very impolite. Besides, there might be other reasons
why certain paths are excluded. They might lead to a large amount of dynamically generated pages
with few differences or worse, to an infinite amount of generated dynamic pages that will leave
Alkaline indexing forever a dataset of a constantly exploding size.
|
