Web services and bulk data

Thread view:

Enable EMail Alerts

Start New Thread

Thread rating:

SB - 29 Oct 2004 12:35 GMT

I am implementing a web services and one of its methods can potentially
return Gb of data. I would like this to be implemented in an 'iterator' kind
of style. With this the client can get the count first and then use a method
to get certain number of records at a time.

My initial idea was to have one web method to return the count of records.
Another web method to return a user spcified number of records and have a
boolean flag to let the client know the last set of records or no more
records. I am thinking of maintaining the state and make this a stateful web
service. I am not sure whether this is the correct way of doing it.

I also heard somewhere about the 'conversational web services'. Is that
something really exist and with .NET?

Signature

Kind regards,
SB

Reply to this Message

Curt Hagenlocher - 29 Oct 2004 19:44 GMT

>I am implementing a web services and one of its methods can potentially
> return Gb of data. I would like this to be implemented in an 'iterator'
> kind
> of style. With this the client can get the count first and then use a
> method
> to get certain number of records at a time.

What Amazon and Google do in their web services is to specify that one page
contains a certain number of records. When you get the first page, you also
get a total page count. You can then request specific page numbers. This
prevents them from having to keep state information.

Naturally, this approach only works well if the underlying data is
relatively static and can be queried appropriately in chunks.
Alternatively, you could cache the results of a query for a particular
length of time, and identify the cached data by matching query parameters.

--
Curt Hagenlocher
curt@hagenlocher.org

Reply to this Message

SB - 02 Nov 2004 11:52 GMT

Hi Curt,

Thanks for your reply. I think data caching is a possibility in my case
along with the record count.

Regards,
SB

> >I am implementing a web services and one of its methods can potentially
> > return Gb of data. I would like this to be implemented in an 'iterator'
[quoted text clipped - 16 lines]
> Curt Hagenlocher
> curt@hagenlocher.org

Reply to this Message

SB - 03 Nov 2004 11:19 GMT

Hi Curt,

Exploring further on the idea of caching the data and providing methods for
the web services client to fetch chunk of the data I am not very sure how
this is achieved.

Assume that the cached data is based on a specific query (filter) from one
consumer application. Obviousely further block of data read requests from
this particular client must be from the cached data that is specific to this
client. How is this usually implemented? Does the server need to tag the
cache with the client uri or something similar? If you know could you give me
bit more info on how google or amazon achieve this kind of scenario?

Thanks,
SB

Reply to this Message

Rate this thread:

Web services and bulk data

Free Magazines