Home | Contact Us | FAQ | Search & Site Map | Link to Us
Sign In | Join | Other 45 Sites in Network
HomeAnnouncementsFree MagazinesWhite PapersSubmit Content
Discussion GroupsASP.NETWindows FormsLanguages.NET FrameworkVisual Studio.NET
Articles.NET FrameworkASP.NETToolsWindows Forms
.NET DirectoryOpen Source ProjectsUser GroupsWeb Resources
Related Topics
Visual Basic 6SQL ServerMS AccessOther DB ProductsMS Server ProductsMore Topics ...

.NET Forum / Languages / C# / March 2008

Tip: Looking for answers? Try searching our database.

Simple Hash algorithm to detect duplicate content

Thread view: 
Enable EMail Alerts  Start New Thread
Thread rating: 
DotNetNewbie - 27 Feb 2008 21:33 GMT
Hello,

I need a simple hash algorithm that will detect duplicate content in
my application.

I want to hash not just the content, but a few other parameters also
like EmployeeID and DepartmentID.

So something like:

int hash = DoHash(string Message, int EmployeeID, int DepartmentID);

Now the hash has to be unique with unique inputs i.e. can't duplicate
the hash value if the inputs are the same.

Ideas?
rossum - 27 Feb 2008 22:12 GMT
>Hello,
>
[quoted text clipped - 10 lines]
>Now the hash has to be unique with unique inputs i.e. can't duplicate
>the hash value if the inputs are the same.
Probably not possible with a reasonable sized hash.  If the size of
the hash is limited, and thare are more than that possible inputs then
there must be some collisions.  How many possible values are there for
the Message string for example?

However it is possible to do something with a reasonably low
probability of a collision, something along the lines of:

int DoHash(string Message, int EmployeeID, int DepartmentID) {
 const int multiplier = 29;
 const int startValue = 37;
 int hash = startValue;
 hash = multiplier * hash + Message.GetHashCode();
 hash = multiplier * hash + EmployeeID;
 hash = multiplier * hash + DepartmentID;
 return hash;
}

This relies on Messsage.GetHashCode() returning a suitable value.
Depending on your exact requirement you may need to put in your own
function there.

Remember that collisions are possible, though they should be rare.  If
you get matching hash values then you must do a full check for
equality.

rossum

>Ideas?
Jon Skeet [C# MVP] - 27 Feb 2008 22:13 GMT
<snip>

> Now the hash has to be unique with unique inputs i.e. can't duplicate
> the hash value if the inputs are the same.
>
> Ideas?

Could you clarify what you mean by the last part? If the inputs are the
same surely the result would *have* to be the same - otherwise it's
useless.

Signature

Jon Skeet - <skeet@pobox.com>
http://www.pobox.com/~skeet   Blog: http://www.msmvps.com/jon.skeet
World class .NET training in the UK: http://iterativetraining.co.uk

DotNetNewbie - 28 Feb 2008 18:38 GMT
> <snip>
>
[quoted text clipped - 10 lines]
> Jon Skeet - <sk...@pobox.com>http://www.pobox.com/~skeet  Blog:http://www.msmvps.com/jon.skeet
> World class .NET training in the UK:http://iterativetraining.co.uk

Sorry I meant "can't duplicate the hash value if the inputs are not
the same".
Jon Skeet [C# MVP] - 28 Feb 2008 18:55 GMT
<snip>

> Sorry I meant "can't duplicate the hash value if the inputs are not
> the same".

In that case you really don't want a hash algorithm. The point of a
hash is to give an *indication* of equality in a reasonably cheap (in
storage and computation) manner.

See http://en.wikipedia.org/wiki/Hash_function

Signature

Jon Skeet - <skeet@pobox.com>
http://www.pobox.com/~skeet   Blog: http://www.msmvps.com/jon.skeet
World class .NET training in the UK: http://iterativetraining.co.uk

Arne Vajhøj - 27 Feb 2008 23:03 GMT
> I need a simple hash algorithm that will detect duplicate content in
> my application.
[quoted text clipped - 8 lines]
> Now the hash has to be unique with unique inputs i.e. can't duplicate
> the hash value if the inputs are the same.

A hash that has to be unique for all input will need to have the same
size as the input meaning that it is useless.

It is really a trade off between risk of collisions with size and
computational effort.

int DoHash(string Message, int EmployeeID, int DepartmentID)
{
    return (Message+EmployeeID+DepartmentID).GetHashCode();
}

only has 2^32 possible values.

string DoHash(string Message, int EmployeeID, int DepartmentID)
{
    MD5 md5 = new MD5CryptoServiceProvider();
    return
Convert.ToBase64String(md5.ComputeHash(Encoding.UTF8.GetBytes(Message+EmployeeID+DepartmentID)));
}

has 2^128 possible values.

Arne
DotNetNewbie - 28 Feb 2008 19:09 GMT
> > I need a simple hash algorithm that will detect duplicate content in
> > my application.
[quoted text clipped - 34 lines]
>
> Arne

Arne, that looks like it is good for me (the string version).
Is that always going to be 32 characters in length?
Jon Skeet [C# MVP] - 28 Feb 2008 19:23 GMT
<snip>

> Arne, that looks like it is good for me (the string version).

It certainly doesn't satisfy your uniqueness constraint. Indeed, we can
even *force* a hash collision without even trying hard.

Consider:

Situation 1:
Message="Hello1"
EmployeeID=2
DepartmentID=3

Situation 2:
Message="Hello"
EmployeeID=12
DepartmentID=3

In both cases you'll be hashing "Hello123".

I still think you'd be better off letting the database work as it's
designed to...

Signature

Jon Skeet - <skeet@pobox.com>
http://www.pobox.com/~skeet   Blog: http://www.msmvps.com/jon.skeet
World class .NET training in the UK: http://iterativetraining.co.uk

DotNetNewbie - 28 Feb 2008 19:57 GMT
> <snip>
>
[quoted text clipped - 23 lines]
> Jon Skeet - <sk...@pobox.com>http://www.pobox.com/~skeet  Blog:http://www.msmvps.com/jon.skeet
> World class .NET training in the UK:http://iterativetraining.co.uk

I see what your saying, then I could do:

string key = String.Format("eid={0}-did={2}-msg={3}", employeeid,
departmentid, message);
then hash the key.

I'm using this for duplicate content checking, not the Article Title
as before John!
Jon Skeet [C# MVP] - 28 Feb 2008 20:25 GMT
> I see what your saying, then I could do:
>
[quoted text clipped - 4 lines]
> I'm using this for duplicate content checking, not the Article Title
> as before John!

Well, you could do that, yes. I still think you should use the database
as it's meant to be used though. You're still not guaranteed to not get
duplicates with this code.

Signature

Jon Skeet - <skeet@pobox.com>
http://www.pobox.com/~skeet   Blog: http://www.msmvps.com/jon.skeet
World class .NET training in the UK: http://iterativetraining.co.uk

DotNetNewbie - 28 Feb 2008 20:44 GMT
> > I see what your saying, then I could do:
>
[quoted text clipped - 12 lines]
> Jon Skeet - <sk...@pobox.com>http://www.pobox.com/~skeet  Blog:http://www.msmvps.com/jon.skeet
> World class .NET training in the UK:http://iterativetraining.co.uk
Jon, how can I check for duplicates using the database (keeping the
employeid/departmentid/message in mind).
Jon Skeet [C# MVP] - 28 Feb 2008 20:53 GMT
> Jon, how can I check for duplicates using the database (keeping the
> employeid/departmentid/message in mind).

Do a query to try to *find* a duplicate - search for an existing record
with the same employeeid/departmentid/message. Alternatively, create a
unique index across the three columns, and just try to insert, knowing
that it'll fail if there's a duplicate.

Signature

Jon Skeet - <skeet@pobox.com>
http://www.pobox.com/~skeet   Blog: http://www.msmvps.com/jon.skeet
World class .NET training in the UK: http://iterativetraining.co.uk

Arne Vajhøj - 28 Feb 2008 23:59 GMT
>> string DoHash(string Message, int EmployeeID, int DepartmentID)
>> {
[quoted text clipped - 8 lines]
> Arne, that looks like it is good for me (the string version).
> Is that always going to be 32 characters in length?

Yes.

Arne
Peter Bromberg [C# MVP] - 28 Feb 2008 03:16 GMT
I am not sure that using a hash computation is the best way to detect
"duplicate content". Are you storing your content in a database? You really
haven't specified much detail. Perhaps you should be looking into a more
robust computation such as CRC32.
-- Peter
Site: http://www.eggheadcafe.com
UnBlog: http://petesbloggerama.blogspot.com
Short Urls & more: http://ittyurl.net

> Hello,
>
[quoted text clipped - 12 lines]
>
> Ideas?
DotNetNewbie - 28 Feb 2008 18:40 GMT
On Feb 27, 10:16 pm, Peter Bromberg [C# MVP]
<pbromb...@yahoo.NoSpamMaam.com> wrote:
> I am not sure that using a hash computation is the best way to detect
> "duplicate content". Are you storing your content in a database? You really
[quoted text clipped - 21 lines]
>
> > Ideas?

Peter,

Yes the content is stored in the database, before anyone inserts new
content I need to check if the same user has posted the same content
before, if he has, then don't insert it again.

Same content means: same employee ID, same departmentID and same
Message.

Meaning that the user can insert the same message text, but it has to
be in a different departmentID.
Jon Skeet [C# MVP] - 28 Feb 2008 18:56 GMT
> Yes the content is stored in the database, before anyone inserts new
> content I need to check if the same user has posted the same content
[quoted text clipped - 5 lines]
> Meaning that the user can insert the same message text, but it has to
> be in a different departmentID.

Right. So you ought to store the employee ID, department ID and message
in the database, and do a query for the combination. So long as all the
columns are indexed, it'll be fine.

Signature

Jon Skeet - <skeet@pobox.com>
http://www.pobox.com/~skeet   Blog: http://www.msmvps.com/jon.skeet
World class .NET training in the UK: http://iterativetraining.co.uk

Rad [Visual C# MVP] - 28 Feb 2008 21:36 GMT
> On Feb 27, 10:16 pm, Peter Bromberg [C# MVP]
> <pbromb...@yahoo.NoSpamMaam.com> wrote:
[quoted text clipped - 11 lines]
> Meaning that the user can insert the same message text, but it has to
> be in a different departmentID.

You could always create a unique constraint constituting the columns you
want to be unique. If the user tries to insert a duplicate the database
will throw an exception that you can trap
Signature

http://www.thinkersroom.com/bytes

Christopher Van Kirk - 29 Feb 2008 04:37 GMT
I'm not a fan of this approach. The message column could be quite
large, and may affect the performance of such an index. Seems like it
would be better to compute a hash of some kind of the message, store
the hashed value in the database, and index on that along with the
other two key fields.

When you want to evaluate whether a particular combo exists, you just
query for the keys and hashed value, then compare your actual message
with the messages you get back from the database.

The weakness of this approach is that if there are millions of
messages for the same employee and department with the same hashed
message value, it could be slow to search through the results of a the
select. It's hard to imagine a scenario in which some employee of some
department could have millions of messages, though.

>> On Feb 27, 10:16 pm, Peter Bromberg [C# MVP]
>> <pbromb...@yahoo.NoSpamMaam.com> wrote:
[quoted text clipped - 15 lines]
>want to be unique. If the user tries to insert a duplicate the database
>will throw an exception that you can trap

Signature

Posted via a free Usenet account from http://www.teranews.com

Jon Skeet [C# MVP] - 29 Feb 2008 07:38 GMT
> I'm not a fan of this approach. The message column could be quite
> large, and may affect the performance of such an index. Seems like it
> would be better to compute a hash of some kind of the message, store
> the hashed value in the database, and index on that along with the
> other two key fields.

But that's exactly what an indexed unique constraint would do, but in a
more transparent fashion.

I've only ever had to manually store a hash in a database once, and
that was to effectively hash an unknown-until-execution-time number of
Guids when populating a set of sets.

Databases know how to index text columns. I think it's best to let them
do their job.

Signature

Jon Skeet - <skeet@pobox.com>
http://www.pobox.com/~skeet   Blog: http://www.msmvps.com/jon.skeet
World class .NET training in the UK: http://iterativetraining.co.uk

DotNetNewbie - 29 Feb 2008 12:33 GMT
> > I'm not a fan of this approach. The message column could be quite
> > large, and may affect the performance of such an index. Seems like it
[quoted text clipped - 15 lines]
> Jon Skeet - <sk...@pobox.com>http://www.pobox.com/~skeet  Blog:http://www.msmvps.com/jon.skeet
> World class .NET training in the UK:http://iterativetraining.co.uk

My message column is NTEXT(MAX), and it is going to have articles in
it.
I'll look into this approach....
Arne Vajhøj - 03 Mar 2008 04:06 GMT
>>> I'm not a fan of this approach. The message column could be quite
>>> large, and may affect the performance of such an index. Seems like it
[quoted text clipped - 14 lines]
> it.
> I'll look into this approach....

You mean NTEXT *or* NVARCHAR(MAX) ?

Well - neither can be indexed by SQLServer ...

Arne
Arne Vajhøj - 03 Mar 2008 04:07 GMT
>> I'm not a fan of this approach. The message column could be quite
>> large, and may affect the performance of such an index. Seems like it
[quoted text clipped - 11 lines]
> Databases know how to index text columns. I think it's best to let them
> do their job.

SQLServer does not.

To quote from BOL:

#Columns that are of the large object (LOB) data types ntext, text,
#varchar(max), nvarchar(max), varbinary(max), xml, or image cannot be
#specified as key columns for an index.

Arne
Jon Skeet [C# MVP] - 03 Mar 2008 07:44 GMT
> > Databases know how to index text columns. I think it's best to let them
> > do their job.
[quoted text clipped - 6 lines]
> #varchar(max), nvarchar(max), varbinary(max), xml, or image cannot be
> #specified as key columns for an index.

That's a pity - and it makes life a bit awkward.

The OP could use a hash and then fetch all values which have the same
hash, then performing the comparison.

Signature

Jon Skeet - <skeet@pobox.com>
http://www.pobox.com/~skeet   Blog: http://www.msmvps.com/jon.skeet
World class .NET training in the UK: http://iterativetraining.co.uk

Christopher Van Kirk - 07 Mar 2008 15:31 GMT
>Arne Vajhoj <arne@vajhoej.dk> wrote:
>> > Databases know how to index text columns. I think it's best to let them
[quoted text clipped - 12 lines]
>The OP could use a hash and then fetch all values which have the same
>hash, then performing the comparison.

Indeed. If you "scroll up" you'll see that this is exactly what I
suggested.

Signature

Posted via a free Usenet account from http://www.teranews.com

Jon Skeet [C# MVP] - 07 Mar 2008 15:43 GMT
> >The OP could use a hash and then fetch all values which have the same
> >hash, then performing the comparison.
>
> Indeed. If you "scroll up" you'll see that this is exactly what I
> suggested.

True - but before we knew that the database couldn't do it all for the
OP in the first place. Where feasible (i.e. not using NTEXT columns
etc) I'd still favour the unique (indexed) constraint in the DB
approach.

Signature

Jon Skeet - <skeet@pobox.com>
http://www.pobox.com/~skeet   Blog: http://www.msmvps.com/jon.skeet
World class .NET training in the UK: http://iterativetraining.co.uk


Rate this thread:







Free Magazines

Get these publications absolutely FREE for up to 12 months. There are no hidden fees and no obligation. Simply choose a title, complete the application form and submit it. Read more ...

Oracle MagazineNetwork ComputingComputer WorldBio-IT WorldeWeekInformation WeekInfosecurity
 
Sign In
Join
My Latest Posts
My Monitored Threads
My Blog
My Photo Gallery
My Profile
My Homepage

Start New Thread
Enable EMail Alerts
Rate this Thread



©2008 Advenet LLC   Privacy Policy - Terms of Use
This website includes both content owned or controlled by Advenet as well as content owned or controlled by third parties.