Chunking Data Accross the Network

A recent project involved sending large PDF documents across several firewalls into an HTTP handler for display.  Unfortunately, the remoting service cannot handle more than 10 Mb of data at a time.  While the most elegant solution may be streaming through a WCF Service, this was not an option at this time.  The second most promising solution as chunking the data across the network.

Here's my implementation, where the server portion sits behind a firewall communicating through a remoting service to a HTTP Handler, which renders a PDF document.  This same technique could be used for any type of file especially considering constraints with File Uploader controls.

Server - ChunkBytes function takes three parameters: document ID (to look up path), Maximum chunk size, and offset position.

ChunkBytes(docID as integer, maxChunk as integer, offset as integer) as Byte()

'first convert file to document stream

Dim documentStream as FileStream = New FileStream(fileLocation, FileMode.Open, FileAccess.Read)

Dim newArray as Byte()

'if the offset position is greater than the length return Nothing

If offset > documentStream.length - 1 then

newArray = Nothing

Else

'create the chunk as a byte array

Dim byteCount as integer

If maxChunk > documentStream.length - offset

'if we're on the last chunk, the array size will be smaller

byteCount = documentStream.length - offset

Else

'otherwise set to maxChunk

byteCount = maxChunk

End If

'create the new array with the correct size

newArray = New Byte(byteCount - 1) {}

'position the file stream to the correct position

documentStream.Seek(offset, SeekOrigin.Begin)

'read the chunk from the filestream into the new array

documentStream.Read(newArray, 0, byteCount)

End If

Return newArray

End Function

The client in our case is on the HTTP Handler inside the ProcessRequest sub.  Each chunk is written directly to the http context, however in another setting you could collect the chunks in another array and output as needed.

'create the maximum chunk size the network can handle we're using 10 Mb

Dim maxChunk as Integer = 10 * 1024 * 1024

Dim offset as Integer = 0

'get the length of the document

Dim documentLength as Integer

'now get the first chunk

Dim documentBytes as Byte() = DocumentByte.ChunkBytes(ID, maxChunk, offset)

'keep getting the next chunk until the function returns nothing

Do Until documentBytes Is Nothing

'write the result to the context

Response.BinaryWrite(documentBytes)

'incriment the offset

offset += maxChunk

'get the next chunk

documentBytes = DocumentByte.ChunkBytes(ID, maxChunk, offset)

Loop