|
 Friday, April 27, 2007

Exposing SQL BLOB Data to MOSS SharePoint 2007

I had been wondering if this was possible as of lately. After not seeing any examples on the web I decided to tackle the task myself.

I wanted to place some files (doc, xls, ppt, htm, txt) into SQL Servers BLOB Field and then expose them to SharePoint 2007 (MOSS). This would allow me to search the BLOB data through the SharePoint seach functionality. Pretty much, I wanted expose a document stored in SQL Server to enable it for searching on MOSS.

A lot of the time applications will store the file path to the file and then store the actual file on a file server. This isnt something I wanted to do, mainly because I was only going to be storing, maybe, 30-50 documents a month. Each of these documents were maybe 1 to 2 pages long, nothing huge. Therefore I decided to keep it with my SQL data, in the datbaase as a BLOB field. This has many advantages, you can keep certain documents with your custom systems. Therefore, if you decide to move the SQL Database later on down the road, you dont have to copy the file share to another server and possibly update the database records with their new location.

So here's what I needed to do:
1. Create a BLOB field in a database table that could be full text searched.
2. Create the Full-Text Index
3. Insert some test documents
4. Create the query to search within the blob documents. (still in sql at this point).
5. Expose the SQL Table information to MOSS through the Business Data Catalog.
6. Allow the content to be searched from within the MOSS site.

After that, WE'RE DONE!

HOW TO DO THIS

 

Blob Table

I created the SQL Table with the following SQL Code:

CREATE TABLE [dbo].[BinaryTable](
   [Row_Id] [int] IDENTITY(1,1) NOT NULL,
   [Blob_Type] [char](3) NOT NULL,
   [Attachment] [varbinary](max) NOT NULL,
   [File_Name] [varchar](100) NULL,
CONSTRAINT [PK_BinaryTable] PRIMARY KEY CLUSTERED 
   (
      [Row_Id] ASC
   )WITH (IGNORE_DUP_KEY = OFF) ON [PRIMARY]
) ON [PRIMARY]

Please note that the Blob_Type is actually the file extension of the document.  SQL Server only supports searching the following BLOBS:
- .doc
- .txt
- .xls
- .ppt
- .htm

From BLOBS in Special Operations in MSDN:

Full-text index and search operations

You can index and search certain types of data stored in BLOB columns. When a database designer decides that a table will contain a BLOB column and the column will participate in a full-text index, the designer must create, in the same table, a separate character-based data column that will hold the file extension of the file in the corresponding BLOB field. During the full-text indexing operation, the full-text service looks at the extensions listed in the character-based column, applies the corresponding filter to interpret the binary data, and extracts the textual information needed for indexing and querying.

When a field in a BLOB column contains documents with one of the following file extensions, the full-text search service uses a filter to interpret the binary data and extract the textual information.
• .doc
• .txt
• .xls
• .ppt
• .htm

The extracted text is indexed and becomes available for querying.

 

Create the Full Text Index


I created a full text catalog.

 


Then I defined the full text index.


The table was now ready for data.

 

Inserting Blob Data.

I followed this example to insert a few documents into the system (just alter the connectionstring and column names. I manually altered it by hand an inserted some documents within a few moments. This was for testing only, in a real world app you'd create a robust method for inserting documents into the system.

The documents I inserted all had greeked text for testing purposes.

 

Querying the Blob.

Once the blob data was inserted, I was then able to query it with this command.

SELECT Row_Id, File_Name FROM BinaryTable WHERE (CONTAINS (Attachment, 'Lorem'))

This searched all the BLOBS for 'Lorem'.

After I got the results I was looking for it was then time to expose it to MOSS.

Exposing the BLOB data to MOSS/SharePoint 2007

In order to expose the data, I had to expose certain metadata entities in the database and certain methods (queries) that allowed SharePoint to understand the search request. This was done through the Business Data Catalog.

This metadata definition had to be built by hand to do enable this functionality. I'm not going to get into how and why things are where they are, all of that info can be found here (expect to spend a good day, or two, to wrap your head around this metadata model).

Here's the code:


BlobSearch.xml

<?xml version="1.0" encoding="utf-8" standalone="yes" ?>
- <LobSystem xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://schemas.microsoft.com/office/2006/03/BusinessDataCatalog BDCMetadata.xsd" Type="Database" Version="1.0.0.6" Name="BlobSearchSample" xmlns="http://schemas.microsoft.com/office/2006/03/BusinessDataCatalog">
- <Properties>
<Property Name="WildcardCharacter" Type="System.String">%</Property>
</Properties>
- <LobSystemInstances>
- <LobSystemInstance Name="BlobSearchSampleInstance">
- <Properties>
<Property Name="AuthenticationMode" Type="System.String">WindowsCredentials</Property>
<Property Name="DatabaseAccessProvider" Type="System.String">SqlServer</Property>
<Property Name="RdbConnection Data Source" Type="System.String">TESTSERVER</Property>
<Property Name="RdbConnection Initial Catalog" Type="System.String">BlobTest</Property>
<Property Name="RdbConnection Integrated Security" Type="System.String">SSPI</Property>
<Property Name="RdbConnection Pooling" Type="System.String">false</Property>
<Property Name="SsoApplicationId" Type="System.String">BlobTest</Property>
<Property Name="SsoProviderImplementation" Type="System.String">Microsoft.SharePoint.Portal.SingleSignon.SpsSsoProvider, Microsoft.SharePoint.Portal.SingleSignon, Version=12.0.0.0, Culture=neutral,PublicKeyToken=71e9bce111e9429c</Property>
</Properties>
</LobSystemInstance>
</LobSystemInstances>
- <Entities>
- <Entity EstimatedInstanceCount="10000" Name="File">
- <Properties>
<Property Name="Title" Type="System.String">File_Name</Property>
</Properties>
- <Identifiers>
<Identifier Name="Row_Id" TypeName="System.Int32" />
</Identifiers>
- <Methods>
- <Method Name="GetFiles">
- <Properties>
<Property Name="RdbCommandText" Type="System.String">SELECT Row_Id, File_Name FROM BinaryTable WHERE (CONTAINS (Attachment, @textToSearchFor))</Property>
<Property Name="RdbCommandType" Type="System.Data.CommandType">Text</Property>
</Properties>
- <FilterDescriptors>
- <!-- Define the filters supported by the back-end method (or sql query) here.
-->
- <FilterDescriptor Type="Wildcard" Name="TextToSearchFor">
- <Properties>
<Property Name="UsedForDisambiguation" Type="System.Boolean">true</Property>
</Properties>
</FilterDescriptor>
</FilterDescriptors>
- <Parameters>
- <Parameter Direction="In" Name="@textToSearchFor">
- <TypeDescriptor TypeName="System.String" AssociatedFilter="TextToSearchFor" Name="TextToSearchFor">
- <DefaultValues>
<DefaultValue MethodInstanceName="FileFinderInstance" Type="System.String">%</DefaultValue>
<DefaultValue MethodInstanceName="FileSpecificFinderInstance" Type="System.String">%</DefaultValue>
</DefaultValues>
</TypeDescriptor>
</Parameter>
- <Parameter Direction="Return" Name="Files">
- <TypeDescriptor TypeName="System.Data.IDataReader, System.Data, Version=2.0.3600.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" IsCollection="true" Name="FileDataReader">
- <TypeDescriptors>
- <TypeDescriptor TypeName="System.Data.IDataRecord, System.Data, Version=2.0.3600.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" Name="FileDataRecord">
- <TypeDescriptors>
<TypeDescriptor TypeName="System.Int32" IdentifierName="Row_Id" Name="Row_Id" />
<TypeDescriptor TypeName="System.String" Name="File_Name" />
</TypeDescriptors>
</TypeDescriptor>
</TypeDescriptors>
</TypeDescriptor>
</Parameter>
</Parameters>
- <MethodInstances>
<MethodInstance Name="FileFinderInstance" Type="Finder" ReturnParameterName="Files" />
<MethodInstance Name="FileSpecificFinderInstance" Type="SpecificFinder" ReturnParameterName="Files" />
</MethodInstances>
</Method>
</Methods>
</Entity>
</Entities>
</LobSystem>


Now that it is exposed I was able to import the Metadata into the BDC on SharePoint, also known as Adding an Application Definition to the Business Data Catalog (click link to learn how to import it into SharePoint)

Searching within SharePoint

Now that the data is defined through the metadata model in SharePoint I can actually search it. I tested this by adding a Business Data List to a page in SharePoint.
I was then able to search for "Lorem" and results came back.

 

I then searched for another key word "Test" that I put in a single document, and that worked as well. See the screen shots below.

 

Conclusion

You could expand this functionality by adding it to the Content Sources and Search Scopes in SharePoint to allow for enterprise search too. But I think I'll save that for another day. :) Or just watch the videos provided in the links.

And that folks, is how you expose BLOB data to SharePoint Search.

 

kick it on SharePointKicks.com
#    Comments [0] |

Flash & 64bit Internet Explorer dont play well together

I run on a 64 bit Vista machine. It has two versions of Internet Explorer installed, a 32 bit and 64 bit. When surfing the web I would notice that sometimes flash ads would display, other times they would not and I'd be prompted to install the flash player. So, I would. BUT... the player would never install.

After finally getting tired of it, I looked into the issue some more. Apparently I thought I was always running the 64 bit version of Internet Explorer, but alas, I was not.

In my task bar, I have an IE Icon that I click on sometimes, and other times I'll fire it up through the command prompt by entering "iexplore".

That was the problem. My Task Bar icon was set to open the 32 bit version of IE, while when I typed "iexplore" into the command window the 64 bit version would run. That would explain why I'm not seeing flash files on my 64 bit version. 64 Bit IE is not supported by flash. Bummer dude.

 

My fix:

I created a shortcut called "ie" in the system32 directory that fires up the 32 bit version (if I need flash support). Which I do when I'm watching episodes of DNRTV.

 

 

#    Comments [0] |
 Thursday, April 26, 2007

Forcing users to provide a message in SVN/TortoiseSVN (Including HOWTO)

I use Subversion (SVN) for my source control and I access SVN through TortoiseSVN so that its integrated into the Windows shell. The past few companies I've worked at have all utilized SVN in one way or another, but we've always had a problem with developers submitting code without attaching a message to the commit.

Why does this matter? If you, as a developer, check in 20 files that are related to project trunk X, without a message, how am I suppossed to know why you checked those into the repository? Did you fix a bug? Did you implement a new feature? Did you complete a task? Did you add a new project to the system? What did you do? Thats the problem if you do not provide a message for each commit, others have no idea of knowing what you did

By not providing the message we're not keeping track of what we did and why we did it. This comes heavily into play when you need to revert back to a certain revision, but you're not sure which revision to go back to. The only thing you remember is "I know it was when we implemented feature 'ABC' into the system." If I provided a comment, I could see in the Tortoise Log that on MM/DD/YYYY at XX:XX time developer "Bob" implemented the feature. Knowing that, I could find the version number and revert back to that revision. Without this information I might spend HOURS UPON HOURS searching through lines of code looking for a key piece that helps me decide if this is the version I want. This could have been avoided if a simple message was provided.

HOWTO: Requiring Messages

By default SVN does not require a message when committing. You can tap into the hooks of SVN and provide some custom code to force your developers to provide a message. The script to do this is below. Note: I cannot take credit for creating this script, as it was found on the net and I dont have the source. So, if you wrote this script, please let me know and I'll add your name here as the author. :) 

Steps:

  1. Create Script
  2. Save as pre-commit.bat
  3. Drop it into the hooks directory of your repository.
  4. DONE

The Script (also provided at the bottom of this post as a download)


@echo off
::
:: Stops commits that have empty log messages.
::

setlocal

set "REPOS=%~1"
set "TXN=%~2"
set "SVNLOOK=c:\progra~1\subversion\bin\svnlook.exe"

:: Make sure that the log message contains some text.
for /f "tokens=*" %%i in ('%SVNLOOK% log -t "%TXN%" "%REPOS%"') do set "LOGMSG=%%i"
if not "%LOGMSG%"=="" exit 0

echo. 1>&2
echo Your commit has been blocked because you didn't give any log message! 1>&2
echo Please write a log message describing the purpose of your changes and 1>&2
echo then try committing again. -- Thank you 1>&2
exit 1


Please note: This assumes that you have Subversion installed in the C:\Program Files\

Save this script as "pre-commit.bat".

Now drop this into your hooks directory of your actual repository. SVN will now execute this batch file each time a commit is taking place. It will check that a message exists. If one does not, it will throw an error message that is highlighted in dark red (above, in the script).

If you are using TortoiseSVN, this is what you will see:

Thats it. Now users cannot commit without a message. :)

 

Good Message Recommendations

Even if messages are required it still does not prevent a user from typing some garbage into the screen. Here's what I recommened to my team...

SVN Messages should contain:

  1. What you did. Give a quick explanation "altered file processing routine" or "created new project library" 
  2. Why you did it. This could be as simple as stating "Fixed Bug 456" or "added new library to handle file conversions."

Example messages:

"Fixed bug 456 by altering file processing routine"

"Added new library to project XYZ to handle file conversions".

There are also many other plug-ins that have been built for TortoiseSVN and SVN in general. Just search for them and you should be able to find something that matches your bug tracking system (such as Gemini).

Conclusion

Forcing the developers to provide a message has enabled the team to communicate better. No longer are members constantly asking the question of "when was this fix applied?" or "What version did this happen in?" The messages make the respository easier to manage and a respository thats easy to manage means we have a healthier code base. :)

Download

pre-commit.zip (.48 KB)

kick it on DotNetKicks.com

#    Comments [12] |
 Wednesday, April 25, 2007

Good SharePoint HowTo Videos

I recently found a couple good videos on SharePoint so I thought I'd share the links.

Microsofts Visual How To's - Videos include, Custom Search Page, Content Sources, Search Scopes, Business Data Catalog, Excel Services and Much more

DNR TV - Videos Include, InfoPath Forms Service, Content Types, Business Data Catalog, and more. Episode 1, Episode 2, Episode 3, Episode 4

Enjoi.

#    Comments [0] |
 Thursday, April 19, 2007

Forcing a download in ASP.NET - How and Why

I really did think this was something every ASP.NET Developer knew how to do. I'd also like to add that I was also under the assumption (which is false) that most all ASP.NET developers know the security risk associated with persisting sensitive (financial, personal, etc) information to a directory available to the web. The example I'm talking about is when a user views a report from their account online and they choose to "export" it for download, this export should NEVER EVER EVER be persisted on the disk (unless you have some major security backing it up or unless its getting deleted right away, but still, I dont recommened it).

I'm going to demonstrate how to force a download, and yes, I'm aware that this has been done by thousands of other sites already. The problem is, apparently there are not enough sites out there that stress the importance of this, so dont flame me for reiterating a supposed well known simple implementation. :) This fits into the same category as "why is your entire business logic in your code behind area", but thats another post, for another day.

The reason is, if we persist a file, lets call it "BobsFinancials.txt" to a directory called "Exports" in the website and then we give Bob a link to download it; he can download it, but so can everyone else. Lets say that "Susy" logs into the system, Susy is Bob's Ex-Girlfriend and she can't stand him, so she has it out for him. Susy knows that Bob uses the same bank as her so she knows that he has the same account screens she does. Susy performs a export for her own account, she see's the url is http://www.example.com/exports/SusysFinancials.txt . She is able to figure out (since she knows that Bob is a user on the system) that Bob's financials might also be in the "exports" folder. So she types in the url: http://www.example.com/exports/BobsFinancials.txt, and blammo, now she has Bobs information.

You'd be surprised how many times I've seen this happen. DOZENS upon DOZENS upon DOZENS. 

So, if we are not to persist the data to the disk, what are we to do? How can we give the data to the user?

HOW TO FORCE A DOWNLOAD AND NOT PERSIST

Well, the data has to come from somewhere, a datastore of some type. You have to build the export somehow (through code). And then you want to give it to the user. Here's how.

1. Buid your export (how ever you want, for example this could be a string of random data)

2. Send it to the user, by forcing a download. We will not be

We've all see it before the dialog box that has a "Save" button present. (Again, I was under the false impression that everyone knew how to do this.) So we need to force a download. In the background whats going on is that the report is stored in memory, we're going to take it from the server's memory and pass it directy to the user through the response stream.

Code

// textExportBuilder - A String Builder that was used to build the report.

sFileName = "BobsFinancials.txt";
Response.ContentType = "application/octet-stream";
Response.AddHeader("Content-Disposition", string.Format("attachment; filename={0}", sFileName));
Response.Write(textExportBuilder);
Response.End();

This code uses the AddHeader method of the Response Object to force the user to download the file. The file has not been saved to the server, it was generated on the fly and given to the user when they needed it and now its gone. There is no actual physical file persisted on the disk that other users can download.

This is a very simple, very beginner thing that A LOT of developers are not aware of.

#    Comments [1] |
 Tuesday, April 17, 2007

Unit Testing .NetTiers Processors with Dependency Injection

Recently my company decided it was time to automate some of the plumbing work, such as the data access layer (thankfully). I checked out a few options (LLBLGen, MyGeneration, CodeSmith and SubSonic) and finally decided on CodeSmith because I had used it before and I felt it was the best fit for the company (in many different areas).

I'm using .NetTiers to create the plumbing code for our applications.

In .NetTiers, Processors are used to add custom business logic into the processing of your data. For example, perhaps you want to make sure that a country only exists ONCE in your system, you could add this functionality here. This thread covers the topic of how to add a a processor to your service layer.

The only problem is that if you want to test your processor using Unit Testing, you are now also testing the data access, database, etc. Its tightly coupled, there's not an easy way to test this. But in actually, there is. Here's how:

The problem lies in the ValidateIfCountry method. It has a dependency on the "CountryService". We cannot mock this out. Its tightly coupled with the Processor. We can eliminate this coupling by injecting a reference to a "CountryService" through constuctor injection. Here's how. (please refer to this thread for full doc on this class).

Previous Constructor Code (with private members shown):

private Entities.Country _country;

/// <summary>
/// Country Processor
/// </summary>
/// <param name="country"></param>
public VerifyCountryProcessor(Entities.Country country)
{
   this._country = country;
}

New Constructor Code, injecting a CountryService:

private Entities.Country _country;
private CountryServiceBase _serviceBase;

/// <summary>
/// Country Processor
/// </summary>
/// <param name="country"></param>
public VerifyCountryProcessor(Entities.Country country, CountryServiceBase serviceBase)
{
   this._country = country;
   this._serviceBase = serviceBase;
}

How to Inject it from the original CountryServiceClass

In the example provided by the .NetTiers team, they said to override the Insert Method with this code, to get the processor to run:

Old Code

public override bool Insert(Order entity)
{
   ProcessorList.Add(new VerifyCountryProcessor(entity));
   Execute();
   return base.Insert(entity);
}

This is is correct, but this time we need to inject a CountryService object into the Processor.

New Code

public override bool Insert(Order entity)
{
   ProcessorList.Add(new VerifyCountryProcessor(entity, this));
   Execute();
   return base.Insert(entity);
}

Since we are already in the "CountryService" class when we're overriding the Insert Method, and since the CountryService inherits from "CountryServiceBase", we can inject this reference into the InventoryProcessor through the constuctor. We have now eliminated the tightly coupled reference inside of the processor. We can now test the functionality through unit testing.

How To Test

I utilize Rhino Mocks for my mocking so here's how to do it with Rhino Mocks.

#region Using Directives
using System;
using System.Collections.Generic;
using System.Text;

using ExampleCompany.App.Entities;
using ExampleCompany.App.Services;
using NUnit.Framework;
using Rhino.Mocks;
using ExampleCompany.App.Services.Pipeline;

#endregion

namespace ExampleCompany.App.UnitTests
{
   [TestFixture]
   public class CountryProcessorTest
   {
      #region Test Setup
      MockRepository mocks;
      CountryService countryService;

      [SetUp]
      public void SetUp()
      {
         mocks = new MockRepository();
         countryService = mocks.CreateMock<CountryService>();
      }


      [TearDown]
      public void TearDown()
      {
         mocks.VerifyAll();
      }

      #endregion 

      #region Tests
      [Test]
      public void TestCountryProcessorForDuplicateName()
      {
         Country country = new Country();
         country.Country_Name = "FOO";

         // Needed for a return value
         TList<Country> cs = new TList<Country>();
         cs.Add(country);

         string whereClause = String.Format("{1} = '{0}'", CountryColumn.Country_Name.ToString(), country.Country_Name);

         Expect.Call(countryService.Find(whereClause)).Return(cs);

         Country country2 = new Country();
         country2.Country_Name = "FOO";

         VerifyCountryProcessor countryProcessor = new VerifyCountryProcessor(countryService, country2);

         Expect.Call(countryService.ToString()).Repeat.Any();

         mocks.ReplayAll();

         IProcessorResult result = countryProcessor.Process();

         Assert.AreEqual(false, result.Result);
         Assert.AreEqual("The country FOO already exists and cannot be added or updated.", result.BrokenRulesLists[typeof(ExampleCompany.App.Entities.Country)][0].Description);
      }


      #endregion
   }
}

Conclusion

By utilizing dependency injection in the processors, we're able to successfully test the processor utiliting unit testing without having to worry about a database connection at the same time. Very helpful. :)

#    Comments [0] |
 Monday, April 16, 2007

Tree Surgeon Development

As of a couple weeks ago, I joined the Tree Surgeon open source project. Recently, Bil Simser took over the project and opened it up on CodePlex. The development team will be adding many new features to the application to help in the setup of the development tree structure and build integration process. Keep your eyes peeled!

#    Comments [0] |
 Friday, April 13, 2007

The .NET Developer Search Engine

While listenening to a recent episode of DotNetRocks, Dan Appleman noted that he created a Google Custom Search Engine called SearchDotNet.com. This search engine searches only relevant .NET sites that Dan himself deems as "experts" in the fields. THis search engine helps with the topic of Discoverability (see the front page of SearchDotNet.com for more info on Discoverability).

Example:

He gave a good example, go to Google.com and search for "RSS".

The top 5 results are:

  1. Rss (wikipedia)
  2. Xml.com (what is Rss)
  3. Rss 2.0 Specification
  4. New York Times Rss
  5. Rdf Site Summary

As a .NET developer this poses no great info for example: how to implement Rss, or maybe .NET Rss feeds.

Enter Dan's search engine, SearchDotNet.com.

Entering the same exact search phrase returns vastly different topics (in regards to .NET).

  1. Scott Gutheries Blog feed
  2. Raymond Chen's blog
  3. Asp.net blogs
  4. Microsoft Team Rss Blog
  5. The ServerSide Rss Feed

Try searching for other topics on his site, such as the word "Triplet". Mads Kristensen blogged about this class awhile back, but searching for "Triplet" on Google.com would return you all kinds of non-.NET related articles.

Dan recommended that the developer try to search on SearchDotNet.com first, then if you cannot find what you're looking for, then go search on Google.com. Thats what I've done and found it immensely useful.

See for yourself, go try it out now.

kick it on DotNetKicks.com
#    Comments [0] |
 Thursday, April 12, 2007

Codeplex is down?

This morning, I went to log into Codeplex and I got the dreaded YSOD. You'd think that Microsoft would have this under wraps. With all of the redundancy and technology that Microsoft has at its fingertips, you'd think this wouldnt be an issue. Sometimes it's great to know, that even the big guys make mistakes. :)  

#    Comments [0] |
 Thursday, April 05, 2007

Keyboard Shortcuts for TestDriven.NET

I'm constantly clicking "Repeat Test Run" or "Run Tests" inside of of the VS 2005 IDE. Yes, I'm willing to admit that I'm a firm believer in going commando when it comes to the mouse. Its just, well, too time consuming. On average, I would normally right-click "Repeat Test Run" or "Run Tests" hundreds of times. This got real old. So I created a keyboard shortcut in Visual Studio.

Some people see me use this and have asked how to do this. Unfortunatley I thought this was a basic thing to know, but apparently its not (at least from the circles that I've run with).

So here's how to do it.

In Visual Studio, go to Tools, Options

Then go to Environment, Keyboard.

Type in "TestDriven" into the Commands Containing window. Select "TestDriven.NET.RunTests" command. Then place your cursor into the "Presss shortcut keys" textbox. Then hit the follow keys: Hold down the CTRL button and then press (while holding down CTRL) R and then release R and then press T (while still HOLDING the CTRL button). This will produce: CTRL + R, CTRL + T. Click assign, and OK and you're done!  

 

Now, inside of your test code, method or class, hit the keyboard shortcut. Hold down CTRL while pressing R and then T. CTRL + R, CTRL + T.

The test will run. You can also apply this to other items such as "Repeat Test Run" and many others. :)

kick it on DotNetKicks.com
#    Comments [0] |
 Sunday, April 01, 2007

Handling unexpected exceptions and tracing

I was browsing a site earlier and I came upon the dreaded Yellow Screen of Death (YSOD). It reminded me of when I had to implement some diagnostic tracing for an application that happened to be built prior to my time at the said company.

The Problem

The application was written in ASP.NET 1.1 and was recently upgraded to 2.0. There was a TON of code in the code behind, heck, we could go as far as to say that the majority of the business logic was in the code behind. The validation and error checking routines were nearly non-existant and as of lately the system was encountering a lot of YSOD screens. The users were confused and on top of that the YSOD screen was presenting TOO MUCH info, enabling a malicious user to take that info and use it for evil purposes. I HAD to get rid of these errors, and quick, and I wasnt allowed to turn off the application so I could fix it. I was allowed to take down the app for a few seconds to implement a change and thats it. I had the error report but could not

I had to implement some way of tracking these bugs without taking down the application for good. I did this through the Application_Error event in the Global.asax file.

Background Info on Page Processing

When a ASP.NET Page is processing and it encounters an error, and if it is not handled, it will present the YSOD with all the pertinent information to the user (that is, if no error handling or default error page is defined in the web.Config). In this project, that was the case. No error handling was present.

An ASP.NET page can also catch its own errors if you implement the Page_Error method. Inside of this method you can get the last error that was thrown and do something with it and then move on. Like this:

public void Page_Error(object sender, EventArgs e)
{
   Exception ex = Server.GetLastError();
   /// Do some stuff with the exception, such as logging, etc 

   // Clean  up 
   Server.ClearError();
}

The problem with this method is that you must implement this on every page. You could do this if had a base page class you inherited from, but what if you forgot to inherit from your base page? Your page would still blow up when something went wrong and was not handled.

Enter Global.asax and Application_Error ...

Implementing Application Level Error Handling & Logging

You can implement application level error handling by implementing some handling at the Application level.

Here's some code, explanation is at the bottom.


Global.asax File

<%@ Application Language="C#" %>
<%@ Import Namespace="System.Web.Configuration" %>

<script runat="server">

void Application_Start(object sender, EventArgs e)
{
}

void Application_End(object sender, EventArgs e)
{
}

void Application_Error(object sender, EventArgs e)
{
   // Perform some handling here if necessary, and then redirect to error handling page as defined in the web.Config file.
   CustomErrorsSection section = (CustomErrorsSection)ConfigurationManager.GetSection("system.web/customErrors");
   Server.Transfer(section.DefaultRedirect);
}

void Session_Start(object sender, EventArgs e)
{
}

void Session_End(object sender, EventArgs e)
{
}

</script>


web.Config File

<?xml version="1.0"?>
<configuration>
    <appSettings/>
    <connectionStrings/>
    <system.web>
         <compilation debug="true"/>
         <authentication mode="Windows"/>
         <customErrors defaultRedirect="Oops.aspx" />
         <trace writeToDiagnosticsTrace="true" enabled="true"/>
    </system.web>
    <system.diagnostics>
        <trace autoflush="true">
            <listeners>
               <add name="TraceSiteEventLogListener" type="System.Diagnostics.EventLogTraceListener,System,version=2.0.0.0,Culture=neutral,PublicKeyToken=b77a5c561934e089" initializeData="Test123"/>
            </listeners>
        </trace>
    </system.diagnostics>
</configuration>


Oops.aspx Page

using System;
using System.Data;
using System.Configuration;
using System.Collections;
using System.Web;
using System.Web.Security;
using System.Web.UI;
using System.Web.UI.WebControls;
using System.Web.UI.WebControls.WebParts;
using System.Web.UI.HtmlControls;
using System.Web.Configuration;

public partial class Oops : System.Web.UI.Page
{
   protected void Page_Load(object sender, EventArgs e)
   {
      Exception ex = Server.GetLastError();
      string errorInfo = GetFormattedException(ex); 
      Trace.Write(errorInfo);
      Server.ClearError(); 
   }

   /// <summary>
   /// Gets the exception information, recursively. 
   /// </summary>
   /// <param name="ex">The exception that occurred.</param>
   /// <returns>A formatted string that contains the message, exception type, source and stack trace for each exception
   /// that exists within the exception hierarchy.</returns>
   private string GetFormattedException(Exception ex)
   {
      string exceptionString = string.Empty;
      // If we have an inner exception, we need to get that information first. 
      if (ex.InnerException != null)
      {
         exceptionString = String.Format("Inner Exception: {0}{1}", GetFormattedException(ex.InnerException), Environment.NewLine);
      }
      string exceptionFormat = "[{0}] - Message: {1} - Exception Type: {2} - Source: {3} - Stack Trace - {4}";
   
      exceptionString += String.Format(exceptionFormat,
            DateTime.Now,
            ex.Message,
            ex.GetType().FullName,
            ex.Source,
            ex.StackTrace);

      return exceptionString;
   }


}



Oops.aspx

<%@ Page Language="C#" AutoEventWireup="true" CodeFile="Oops.aspx.cs" Inherits="Oops" %>

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<html xmlns="http://www.w3.org/1999/xhtml" >
   <head runat="server">
      <title>Error Page</title>
   </head>
<body>
   <form id="form1" runat="server">
   <div>
      Oops! An error occurred!
   </div>
   </form>
</body>
</html>



Explanation

If an error is thrown, on any page, the Global.asax's Application_Error event catches the error. At that time I redirect the user to the default error page that is defined in the web.Config's customErrors section (Oops.aspx).

In that file (Oops.aspx) I grab the last exception by making a call to Server.GetLastError(). Then I recursively loop through each child exception that the exception has and format the data for viewing purposes.

After the data is formatted I then write it to the Page.Trace, which is set up to write it to the Trace Listeners that are set up in the web.config. I have decided to use the Event Log as my location of writing my trace information to, but this could easily be set up to use the TextWriterTraceListener (or any other listener) as well.

Then at the end, I clean up the exception queue by calling Server.ClearError().

Then, the page is displayed. "Oops! An error occurred!".

Now, open up the event viewer and look inside of the application log. You'll see a source by the name of "Test123" and you'll see the error information inside. You'll also find a plethora of other great event information too.  

For this to all work, we have to set the Trace enabled to true in the web.Config and set the writeToDiagnosticsTrace equal to true as well. Otherwise the Page's Trace will not redirect its trace information to the listeners defined in the web.Config. To turn this off, just set the enabled flag to "false".

Conclusion

The best thing to do is to always capture the error, give the user a meaningfull message such as "First name cannot be blank." But at times there are errors that we don't catch as developers, things slip through the cracks and we miss something, that's life, it happens. In event of such case, we can enable global exception handling, just like we did here. Just in case something "exceptional" happens, we can catch it, log it in the event viewer and not have to worry about displaying vital information to a user through the YSOD.

 

#    Comments [0] |