MD5 Message-Digest Algorithm is described in RFC 1321 available at http://www.faqs.org/rfcs/rfc1321.html . The algorithm takes as input a message of arbitrary length and produces as output a 128-bit fingerprint or message digest of the input. It is conjectured that it is computationally infeasible to produce two messages having the same message digest, or to produce any message having a given pre-specified target message digest.
The MD5 digest mechanism within Alkaline enables removal of duplicate pages pointed by two different urls. This is typical for documents that can be both accessed with http://www.server.com/ and http://www.server.com/index.html . Alkaline believes that this is a different document and catches the fact that they are identical by comparing the MD5 digests.
Setting this option to N when indexing single urls from different sites or fully dynamic pages may speed-up the indexing process.
The NoMd5 option with exact opposite effect was deprecated in version 1.6.0824.0 and replaced by Md5 . The deprecated option is still supported by the engine.