Chunking Data Accross the Network

A recent project involved sending large PDF documents across several firewalls into an HTTP handler for display.  Unfortunately, the remoting service cannot handle more than 10 Mb of data at a time.  While the most elegant solution may be streaming through a WCF Service, this was not an option at this time.  The second most promising solution as chunking the data across the network.

Here's my implementation, where the server portion sits behind a firewall communicating through a remoting service to a HTTP Handler, which renders a PDF document.  This same technique could be used for any type of file especially considering constraints with File Uploader controls.

Server - ChunkBytes function takes three parameters: document ID (to look up path), Maximum chunk size, and offset position.

ChunkBytes(docID as integer, maxChunk as integer, offset as integer) as Byte()

'first convert file to document stream

Dim documentStream as FileStream = New FileStream(fileLocation, FileMode.Open, FileAccess.Read)

Dim newArray as Byte()

'if the offset position is greater than the length return Nothing

If offset > documentStream.length - 1 then

newArray = Nothing

Else

'create the chunk as a byte array

Dim byteCount as integer

If maxChunk > documentStream.length - offset

'if we're on the last chunk, the array size will be smaller

byteCount = documentStream.length - offset

Else

'otherwise set to maxChunk

byteCount = maxChunk

End If

'create the new array with the correct size

newArray = New Byte(byteCount - 1) {}

'position the file stream to the correct position

documentStream.Seek(offset, SeekOrigin.Begin)

'read the chunk from the filestream into the new array

documentStream.Read(newArray, 0, byteCount)

End If

Return newArray

End Function

The client in our case is on the HTTP Handler inside the ProcessRequest sub.  Each chunk is written directly to the http context, however in another setting you could collect the chunks in another array and output as needed.

'create the maximum chunk size the network can handle we're using 10 Mb

Dim maxChunk as Integer = 10 * 1024 * 1024

Dim offset as Integer = 0

'get the length of the document

Dim documentLength as Integer

'now get the first chunk

Dim documentBytes as Byte() = DocumentByte.ChunkBytes(ID, maxChunk, offset)

'keep getting the next chunk until the function returns nothing

Do Until documentBytes Is Nothing

'write the result to the context

Response.BinaryWrite(documentBytes)

'incriment the offset

offset += maxChunk

'get the next chunk

documentBytes = DocumentByte.ChunkBytes(ID, maxChunk, offset)

Loop

Image Rendering: Bicubic Interpolation

This article shows a quick fix for IE7 image rendering.  The CSS fix applies bicubic interpolation algorithm for image rendering.  I used it like this,     img {-ms-interpolation-mode:bicubic;}

I honestly had to look up this wiki article regarding bicubic interpolation.  

The first image demonstrates a bicubic interpolation implementation.

While the second image demonstrates a "nearest-neighbor" interpolation on the same data set.

Silverlight Toolkit

While I've built a simple Silverlight control, we didn't get to use it because they don't want customers to have to download the Silverlight.  

However, this could be very cool and an easier use of the technology with the Silverlight toolkit.  I use the Ajax toolkit everyday, I'm sure this one could be just as helpful.

Design Process

Great article in the Make blog regarding the design process.  Notice the loop in re-design and test/evaluate phase.  In my opinion, this iteration will consume the vast majority of development time.  We often refer to it as the refactoring stage.

This refactoring iteration is also the step which many project planners fail to allocate enough time for in their planning.

Centering With CSS

Seems like this is a well known CSS technique.  May need it in the future.

#Content

{

margin: 0 auto;

text-align: left;

}

Cross Browser Compatibility - Hay Caramba

Recently reminded of the fun debugging a public application to be compatible in many browsers. We currently test for IE6, IE7, Safari, and Firefox.

Interestingly, the most difficult to debug is IE6, due to the inability to download both 6 and 7. Take a look at this IE6 Standalone application, which helped tremendously.

Also, don't forget javascript functions to determine which browser the visitor is using.

YAHOO 404 Fun Picture

Believe this shot is from Giants Stadium (AT&T Ballpark). Definitely an unfortunate ad placement.

Minimize Coupling, Maximize Cohesion

Understanding this maxim, also explains some of the reasoning behind N-tier architecture and object oriented development techniques.

In fact, the CSLA framework forces minimal coupling due to the ability to remotely access various layers in the application accross many servers.

Scrolling Tab Content

Check out this vertically or horizontally scrolling tab control. If you use VS 2008's ListView, you should be able to adjust the content dynamically.

Web Site Design Fun

View the source on this collection of Div tags and find that there is no img tags, only some very impressive work with HTML div tags, and inline styling.

Who Else Hand Codes HTML and CSS

My most recent project is constructing a public web application for the Historical Legal Documents kept by the Harris County District Clerk. While writing this public application, it is the first time I've put much thought into cross-browser compatibility, proving to be an interesting challenge.

In this project and others, I've often debated the merits of learning to hand code HTML instead of relying on a WSIWYG visual designers. Saying this, I was happily surprised to see this Q&A from the designer of the NY Times website, Khoi Vinh. They commented that they use text editors to design the site.

It’s our preference to use a text editor, like HomeSite, TextPad or TextMate, to
“hand code” everything, rather than to use a wysiwyg (what you see is what you
get) HTML and CSS authoring program, like Dreamweaver. We just find it yields
better and faster results.

As a hand coding side note, I would also fervently encourage learning to hand code SQL. I find complex queries and roll out scripts would either be severely crippled or not possible without the hand coded SQL knowledge.

Reading and Programming

Coding Horror narrows down a programming reading list to five. Many apply to software analysis as well as to writing code.

1. Code Complete 2
2. Don't Make Me Think
3. Peopleware
4. Pragmatic Programmer
5. Facts and Fallacies

In addition to reading, I feel writing is important also. Preparing a blog entry seems to provide a great learning opportunity similar to teaching.

ViewState versus SessionState

I had previously adopted a practice of using sessionstate to persist data across the stateless web. Unfortunately, this practice carried many unanticipated consequences (such as timing out frequently).

I was recently introduced to the viewstate object, which stores variables in a similar hash-table like configuration. The significant benefit being that the viewstate is stored on the client, and will not timeout as long as the page remains open. Other benefits include the ability to expand to a web-farm web-server arrangement (this is not possible with a server stored variable), and a reduction in memory bloat on the web-server.

There are, of course, drawbacks. The most significant is page performance. This is because the viewstate variables are streamed to the client. This drawback should be reduced with the partial postbacks in the AJAX architecture.

Here is a simple example of storing a viewstate object. Remeber that like the sessionstate, the view state can store any serializable object:

ViewState("tagName") = "this Tag Name"

Dim thisTagName as String = ViewState("tagName")
Response.Write(thisTagName)

New Google App Engine

Interesting comments regarding the announcement of the new Google App Engine (it even has an IDE) from the inventor of RSS. His comment at the end is inspiration for new learning, "Python is the new Basic". Time to hit the books again.

New Blog Title

Test new RSS feed for new blog. Simplicy is not simple

Getting Data Into Silverlight

Having good success with the pre-built Silverlight control SlideShow, implementing a custom http handler (ashx) to serve the control with dynamic XML data. However, the missing ingredient in all of the basic Silverlight demonstrations is examples of how to fill the client-side control with server side data.

Here are two demonstrations of the recommended WCF technique. I haven't tried them out yet, and the decision makers still require Microsoft to move this software out of Beta before having public web apps which use it. However, better to get a head start, because this stuff will improve usability significantly.

Not sure if the Maersk guys are still reading this feed, but hope all is well with you. Hear you might have to wait a little longer for your 2007 performance bonus. Good luck to you guys. Probably need to adjust the name of this feed, but will leave the address the same. I don't guess anyone is reading anyway, just a diary for my ideas.

Are Video Games Evil, and Are We There Yet?

What we found with Isaac once, is let him design his own video game.

Always Simplify User Interface

I found myself struggling with the design of a new project until finding this cartoon. Must remember the most successful comercial designs have a over-simplified interface.

Likewise, let the simplicity of the design bleed over into the database tier. Consider implimenting hash-table like key-value table designs. This will facilitate the search type interface with a fast, flexible, tag-based backbone.

Zen Quote of the Day

Here's an interesting quote of the day for our aspiring students.

There are grammatical errors even in his silence. Stanislaw J. Lec

Custom Image Control

While creating a custom document reader for the historical document project, I required an image control which will allow conversion of tiff files to jpeg. The default .NET image control does not render tiff images, and will only allow output directly from an ImageURL property. This restriction limits a simple interface for the image control when image manipulation is necessary before rendering, or when embedding the image control in a data bound list type control.

With inspiration from this control designer, Peter Bromberg, I have re-written his custom image control in visual basic and added inheritance directly from the asp:image control. The custom control has a bitmap property set at run time, outputting a Jpeg image directly to the control. As noted, this additional functionality enables the image control to receive an image directly from a data bound list control, and allows any sort of server side image conversion before rendering.

Here are the two classes used to compile the new control (ImageControl, ImageHandler). At runtime, any GDI+ image conversion is done on the server, then the image is set to the bitmap property of the control.

While progressing with this project, I hope to encapsulate this custom control inside of a Visual Studio 2008 ListView control, which is bound to a database or XML datasource. This will eventually make a document viewer for historical documents.

As Scott Guthrie says, "Happy Coding!"

Big Retirement Announcements

Not sure you guys are still reading the blogs. However, hear there were significant retirement announcements at the HRC this week. Any comments?

Hope all is well, and Happy Chinese New Year!

My email by the way is, mhrugby@hotmail.com

Thoughts on Learning

I have collected these quotes while at Maersk learning software development. Perhaps they are of little use, but many have helped me apreciate learning.

Troy asked me to think about the process which leads to successful problem solving. Certainly a difficult concept to define, but maybe summed up by the phrase I enjoy saying, “be creative, be innovative.”

Being creative and innovative is certainly easier said than done; however, the best path I’ve found to this is having a wiliness to learn, and be willing to learn from the most unexpected places.

• Regarding Buckminster Fuller -- The first prerequisite for continued education is a receptiveness to one's environment. Calluses worn through a faulty environment dull the learning senses. We must educate ourselves to do more with less in creating a suitable environment. Vicious circle? Nonsense: Fuller tells us that no curve can overlap with itself! This is an upward spiral into which Fuller propels us. There is no alternative.

• "To me, there are three things we all should do every day. We should do this every day of our lives. Number one is laugh. You should laugh every day. Number two is think. You should spend some time in thought. Number three is, you should have your emotions moved to tears, could be happiness or joy." Jim Valvano

• Demonstrate an attitude that leads to competence -- be an active partner in developing a solution.

• "Programming today is a race between software engineers striving to build bigger and better idiot-proof programs, and the Universe trying to produce bigger and better idiots. So far, the Universe is winning." Rich Cook

• It's so much easier to suggest solutions when you don't know too much about the problem.
  Malcolm Forbes (1919 - 1990)

• Technology is dominated by two types of people: those who understand what they do not manage, and those who manage what they do not understand.
  Putt's Law

• The most likely way for the world to be destroyed, most experts agree, is by accident. That's where we come in; we're computer professionals. We cause accidents.
  Nathaniel Borenstein (1957 - )

• A man is not idle because he is absorbed in thought. There is a visible labor and there is an invisible labor. Victor Hugo (1802 - 1885)

• A supervisor must be willing to look their employee in the eye for a successful relationship.

• Moore's Law -- Another, sometimes misunderstood, point is that exponentially improved hardware does not necessarily imply exponentially improved software to go with it. The productivity of software developers most assuredly does not increase exponentially with the improvement in hardware, but by most measures has increased only slowly and fitfully over the decades.

• To err is human, but to really foul things up requires a computer.
  Farmers' Almanac, 1978

• "The only people for me are the mad ones, the ones who are mad to live, mad to talk, mad to be saved, desirous of everything at the same time, the ones who never yawn or say a commonplace thing, but burn, burn, burn, burn like fabulous roman candles exploding like spiders across the stars…" Jack Kerouac

• One ought, every day at least, to hear a little song, read a good poem, see a fine picture, and if it were possible, to speak a few reasonable words.
  Johann Wolfgang von Goethe (1749 - 1832)

• So you see, imagination needs moodling - long, inefficient, happy idling, dawdling and puttering. Brenda Ueland

Quotes of the Day

Interesting quotes for the day. The first I think is very important when considering how a business deals with its employees, and how a supervisor deals with their team. Both I think are important when considering

Anger at lies lasts forever. Anger at truth can't last.
Greg Evans

You can't build a reputation on what you are going to do.
Henry Ford

SQL: Database Normalization

Technique for organizing relational databases to minimize duplication of data. While performing this minimizing function, the technique also prevents certain logical or structural data anomalies.

Higher degrees of normalization tend to involve the creation of more tables, and more joins when retrieving functional datasets.

Data anomaly types addressed by normalization

• Update Anomaly - similar data is represented in multiple areas. When data is updated, it may not capture each specific source. When queried, this results in multiple records and inconsistent results.
• Insertion Anomaly - When combined with the incorrect combination of fields, an insert statement into the table may be incorrectly refused.
• Deletion Anomaly - Similar to insertion, when combined incorrectly on a table, the last instance of a record may be incorrectly deleted.

The normal forms of a database determine the degree of vulnerability to data inconsistencies. The higher the form, the less vulnerable the table is.

• First Normal Form (1NF) - The table must faithfully represent a relation. One requirement for a relation being that there is only one value for each attribute or "no repeating groups"
• Second Normal Form (2NF) -
• Third Normal Form (3NF) -

Purposeful denormalization can be used to represent by read-only applications or Business Intelligence applications. These are characterized as OLAP (Online Analytical Processing). An implementation of this is the Amazon database previously written about, allowing data set in a field, more like hash table/key value pairs.

SI Acknowledgement Notifications

Initial development has been completed for these notifications. The test service is currently only available on my desktop. This is a pilot windows service, which currently sends notices to customers (the application will send the notice first the forwarder, if no forwarder then the shipper) after receiving an initial MASTER B/L, MASTER SED, or EDI MASTER Image type from MDWS.

The solution has been added to Source Safe as SIAcknowledgementSolution. Here are the sample notifications (Maersk and Safmarine). I went ahead and made the service so it will handle Safmarine bills, even though they haven’t asked for it.

Testing the implementation, as with most windows services is not easy, but I think we can come up with something. Basically, we need to fill the base table, then run the service from my desk.

1. Database – the initial data is generated from the MDWSProcessImageLink stored procedure, inserting data into the SIAcknowledgementBillTable when the ‘Master B/L’ criteria is met.
   
   • Tables

• SIAcknowledgementBillTable – holds initial data from MDWSProcessImageLink
• SIAcknowledgementReportTable – holds records which a Crystal Report notification is processing from
• SIAcknowledgementContactTable – holds the pilot group customer numbers and contact numbers

Stored Procedures

• SIAcknowledgementGetBillNumber – pulls the initial set of data for the application.
• SIAcknowledgementReportSproc – moves the appropriate data into the report tables
• SIAcknowledgementIndex – indexes the data sent in the notification. The data is indexed into the Images, CustomerInteractions, and CustomerInteractionReasons.

File Storage - Files will be stored on the CDTStore drive.

Application
a.Location – Assume we will install this on the SCRBCDTUSWOFCL2 server
b. Running Time – The requirements say as real-time as possible, I’m thinking about putting the timer at 10 or 20 minutes. This may accomplish close enough to real time as necessary for the pilot group.
c.Who to send to – Send notice first to Forwarder, if no forwarder, then shipper
d.Maersk or Safmarine – compares line code to find company

How Much Does An MBA Cost

Interesting article discussing the relative costs versus benefits for an MBA. They conclude that the most significant benefit for an MBA is the network of friends you develop.

I think I would go for the self taught method.

Import Inland Arrival Notices - Discharge/Clearance Port Logic

The import department has requested a review of the logic necessary to send inland arrival notices. We have developed a new set of criteria which should simply the process significantly.

The new logic follows below, and relies significantly on the comparison between the discharge and the clearance port (this is a new field in our ACMain table).

1. If Discharge/Clearance match, then send notice when container gates in or off-rails at delivery location.
2. If Discharge/Clearance do not match.
• Send notice when container gates in or off-rails at clearance port
• Send notice when container gates in or off-rails at delivery location

Here are the objects adjusted to enable the change in logic, most of this was done to accommodate the new criteria of Discharge/Clearance match.

1. Tables
1. ACMain –
• DischargeClearanceMatchGroup field added
• Adjusted Process Steps 70, 71, 72 to hold new filters
• Deleted Process Steps 73, 74, and 75 which are no longer necessary
2. ACDischargeClearanceMatchGroups – new table
3. ACDischargeClearanceMatchValues – new table
2. Views
1. ACShipmentDataView – added joins and field (DischargeClearanceMatch)
2. ACProcessingView – added joins and field for new logic
3. Stored Procedures
1. ACShipmentDataSproc – This is the main stored procedure, which needed new variables to hold the discharge/clearance field
2. ACLogicSproc – This stored procedure compares the activity to the ACMain logic table.