Home | Contact Us | FAQ | Search & Site Map | Link to Us
Sign In | Join | Other 45 Sites in Network
HomeAnnouncementsFree MagazinesWhite PapersSubmit Content
Discussion GroupsASP.NETWindows FormsLanguages.NET FrameworkVisual Studio.NET
Articles.NET FrameworkASP.NETToolsWindows Forms
.NET DirectoryOpen Source ProjectsUser GroupsWeb Resources
Related Topics
Visual Basic 6SQL ServerMS AccessOther DB ProductsMS Server ProductsMore Topics ...

.NET Forum / Languages / C# / May 2008

Tip: Looking for answers? Try searching our database.

Performance to convert byte[] to a value

Thread view: 
Enable EMail Alerts  Start New Thread
Thread rating: 
Roy - 04 May 2008 03:50 GMT
What is the way to have best performance to copy a byte[] to a value such as
long?
I use
BitConverter.ToInt64(binary, offset)
But the performace is not good enough. I need to have the best performance
in my case.
Arne Vajhøj - 04 May 2008 04:23 GMT
> What is the way to have best performance to copy a byte[] to a value such as
> long?
> I use
> BitConverter.ToInt64(binary, offset)
> But the performace is not good enough. I need to have the best performance
> in my case.

Do you know something about the bytes you will have that makes them
special compared to the general case Microsoft has coded for ?

If not I find it difficult to believe that it will be possible
to find something faster.

Oh - and are you sure this is really the bottleneck in your app ??

(you should be able to do 100-200 million conversions per second)

Arne
Aneesh Pulukkul [http://dotnet-revolutions.blogspot.com] - 04 May 2008 12:42 GMT
> > What is the way to have best performance to copy a byte[] to a value such as
> > long?
[quoted text clipped - 14 lines]
>
> Arne

As per my understanding BIrConverter is the apt option to convert byte
array to numeric. It has a property to check the endianness also -
IsLittleEndian. Do not know how bad it is from perdormance point of
view. Do you have any benchmark results?
Arne Vajhøj - 04 May 2008 16:41 GMT
>>> What is the way to have best performance to copy a byte[] to a value such as
>>> long?
[quoted text clipped - 16 lines]
> IsLittleEndian. Do not know how bad it is from perdormance point of
> view. Do you have any benchmark results?

The endianess check is a single if statement (I looked with reflector).

It uses shift and or to create the long.

I can not right away come up with any ideas that I would expect
to be faster.

I did test BitConverter against a union (struct with field offset) and
BitConverter was much faster.

If it was a huge array that needed to be converted, then possible
a call to native code could be worth considering.

Arne
Roy - 04 May 2008 17:27 GMT
I do have a List<byte[]> with one million of byte[] entries (each of tem has
3 values) to convert. It takes 1 - 2 seconds to convert. And it is too slow
for my case. What is the native code to convert it with faster performance? I
don't mind how ugly the code looks like as long as the performance is best in
my case.

> >>> What is the way to have best performance to copy a byte[] to a value such as
> >>> long?
[quoted text clipped - 31 lines]
>
> Arne
Jeroen Mostert - 04 May 2008 21:43 GMT
> I do have a List<byte[]> with one million of byte[] entries (each of tem has
> 3 values) to convert. It takes 1 - 2 seconds to convert. And it is too slow
> for my case. What is the native code to convert it with faster performance? I
> don't mind how ugly the code looks like as long as the performance is best in
> my case.

How exactly do you *obtain* a list with one million byte arrays? Wouldn't it
be much faster (and more memory-efficient) to change whatever is producing
such a list to produce the list you need in the first place?

What's the result of the conversion? Another list? Keep in mind that
allocating memory takes time too. The conversion of individual entries is
not the bottleneck, in the sense that you will not be able to speed that up
much. You'll want to focus on minimizing the conversion effort.

Signature

J.
http://symbolsprose.blogspot.com

Roy - 04 May 2008 22:44 GMT
I have a basic item consisting of 3 values (I know the types and they alway
together). An item consisted of any number of the basic items can be stored
as a record in database as varbinary(max). After I retrieve the records
(items) from database, I need to
(1) split the record (item) into base items (because the client want to see
the basic item)
(2) restore each basic item from the binary to the original types (3 values)
And I have about 1 million basic items. That's why the performance is very
important for me. I stored multiple basic items together as an item
(varbinary(max)) in database to improve the performace for query. Now I need
to improve the performace for (1) and (2). Any suggestions?
Thanks.

> > I do have a List<byte[]> with one million of byte[] entries (each of tem has
> > 3 values) to convert. It takes 1 - 2 seconds to convert. And it is too slow
[quoted text clipped - 10 lines]
> not the bottleneck, in the sense that you will not be able to speed that up
> much. You'll want to focus on minimizing the conversion effort.
Jon Skeet [C# MVP] - 04 May 2008 22:50 GMT
> I have a basic item consisting of 3 values (I know the types and they alway
> together). An item consisted of any number of the basic items can be stored
[quoted text clipped - 8 lines]
> to improve the performace for (1) and (2). Any suggestions?
> Thanks.

As Arne said, BitConverter is already very fast - on my laptop it goes
through 100 million arrays in a second.

How long is it taking you to get the data from the database in the
first place? I'd expect that to utterly dwarf the conversion time.

Signature

Jon Skeet - <skeet@pobox.com>
Web site: http://www.pobox.com/~skeet   
Blog: http://www.msmvps.com/jon.skeet
C# in Depth: http://csharpindepth.com

Arne Vajhøj - 05 May 2008 03:38 GMT
> I do have a List<byte[]> with one million of byte[] entries (each of tem has
> 3 values) to convert. It takes 1 - 2 seconds to convert. And it is too slow
> for my case. What is the native code to convert it with faster performance? I
> don't mind how ugly the code looks like as long as the performance is best in
> my case.

On my PC it takes about 0.043 seconds to convert each byte[3] to long
in a list with one million elements.

Actually I had to test with ten million elements and divide by 10,
because I could not really measure with so small data as one
million elements.

Your 1-2 seconds must be spend on something else.

I think you need to analyze the app again.

List<byte[]> is not suitable for native code I think.

But because there are just 3 bytes you may be able to
do it faster that BitConverter that has to handle all
8 bytes.

The 0.043 seconds are with:

        private static long Cvt1(byte[] b)
        {
            byte[] tmp = new byte[8];
            tmp[0] = b[0];
            tmp[1] = b[1];
            tmp[2] = b[2];
            return BitConverter.ToInt64(tmp, 0);
        }

but using:

        private static long Cvt2(byte[] b)
        {
            return (b[2] << 16) | (b[1] << 8) | b[0];
        }

then I can get it down to 0.006 seconds.

But if it is probably not relevant. If your existing code takes 1.000
seconds then the difference will just reduce it to 0.963 seconds - noone
will notice the difference.

Arne
Ben Voigt [C++ MVP] - 05 May 2008 15:17 GMT
>> I do have a List<byte[]> with one million of byte[] entries (each of tem
>> has 3 values) to convert. It takes 1 - 2 seconds to convert. And it is
>> too slow for my case. What is the native code to convert it with faster
>> performance? I don't mind how ugly the code looks like as long as the
>> performance is best in my case.

How are you iterating through the list?  Are you using any virtual calls
(through an IList or IEnumerable interface, for example)?  Retrieving an
item from the List is probably several times slower than the BitConverter
call.

> On my PC it takes about 0.043 seconds to convert each byte[3] to long
> in a list with one million elements.

What?  According to the code for ToInt64, that would throw.

 if (startIndex > (value.Length - 8))
   {
       ThrowHelper.ThrowArgumentException(ExceptionResource.Arg_ArrayPlusOffTooSmall);
   }

> Actually I had to test with ten million elements and divide by 10,
> because I could not really measure with so small data as one
[quoted text clipped - 9 lines]
> do it faster that BitConverter that has to handle all
> 8 bytes.

To start with, just use ToInt32 instead...

> The 0.043 seconds are with:
>
[quoted text clipped - 6 lines]
> return BitConverter.ToInt64(tmp, 0);
> }

Ahh, that's not a byte[3] after all.

> but using:
>
[quoted text clipped - 10 lines]
>
> Arne
Arne Vajhøj - 06 May 2008 02:03 GMT
>>> I do have a List<byte[]> with one million of byte[] entries (each of tem
>>> has 3 values) to convert. It takes 1 - 2 seconds to convert. And it is
[quoted text clipped - 6 lines]
> item from the List is probably several times slower than the BitConverter
> call.

I would expect retrieving a ref to the byte[] from a List<byte[]> to
be much faster than BitCoverter.ToInt64, but I have not measured it
and I can be completely wrong.

>> On my PC it takes about 0.043 seconds to convert each byte[3] to long
>> in a list with one million elements.
[quoted text clipped - 5 lines]
>         ThrowHelper.ThrowArgumentException(ExceptionResource.Arg_ArrayPlusOffTooSmall);
>     }

>> The 0.043 seconds are with:
>>
[quoted text clipped - 8 lines]
>
> Ahh, that's not a byte[3] after all.

Let us just say that I got an exception in first attempt ...

:-)

Arne
Ben Voigt [C++ MVP] - 08 May 2008 15:05 GMT
>>>> I do have a List<byte[]> with one million of byte[] entries (each
>>>> of tem has 3 values) to convert. It takes 1 - 2 seconds to
[quoted text clipped - 11 lines]
> be much faster than BitCoverter.ToInt64, but I have not measured it
> and I can be completely wrong.

BitConverter.ToInt64 ought to be much faster than a virtual call, because it
will be inlined and there is no need for a pipeline flush.

In fact, unless you're dealing with a non-native byte order,
BitConverter.ToInt64 should reduce to just a 64-bit load.

I wouldn't be surprised to learn that the JIT generates considerably less
efficient code, though.
Arne Vajhøj - 12 May 2008 01:40 GMT
>>> How are you iterating through the list?  Are you using any virtual
>>> calls (through an IList or IEnumerable interface, for example)?
[quoted text clipped - 12 lines]
> I wouldn't be surprised to learn that the JIT generates considerably less
> efficient code, though.

It is not just the shift's and or's. It is also all the if's.

Arne

Rate this thread:







Free Magazines

Get these publications absolutely FREE for up to 12 months. There are no hidden fees and no obligation. Simply choose a title, complete the application form and submit it. Read more ...

Oracle MagazineNetwork ComputingComputer WorldBio-IT WorldeWeekInformation WeekInfosecurity
 
Sign In
Join
My Latest Posts
My Monitored Threads
My Blog
My Photo Gallery
My Profile
My Homepage

Start New Thread
Enable EMail Alerts
Rate this Thread



©2008 Advenet LLC   Privacy Policy - Terms of Use
This website includes both content owned or controlled by Advenet as well as content owned or controlled by third parties.