Technical Solutions

Applications of AI

2008-02-03T07:09:00.002-08:00

Q. What are the applications of AI?
A. Here are some.
game playing
You can buy machines that can play master level chess for a few hundred dollars. There is some AI in them, but they play well against people mainly through brute force computation--looking at hundreds of thousands of positions. To beat a world champion by brute force and known reliable heuristics requires being able to look at 200 million positions per second.
speech recognition
In the 1990s, computer speech recognition reached a practical level for limited purposes. Thus United Airlines has replaced its keyboard tree for flight information by a system using speech recognition of flight numbers and city names. It is quite convenient. On the the other hand, while it is possible to instruct some computers using speech, most users have gone back to the keyboard and the mouse as still more convenient.
understanding natural language
Just getting a sequence of words into a computer is not enough. Parsing sentences is not enough either. The computer has to be provided with an understanding of the domain the text is about, and this is presently possible only for very limited domains.
computer vision
The world is composed of three-dimensional objects, but the inputs to the human eye and computers' TV cameras are two dimensional. Some useful programs can work solely in two dimensions, but full computer vision requires partial three-dimensional information that is not just a set of two-dimensional views. At present there are only limited ways of representing three-dimensional information directly, and they are not as good as what humans evidently use.
expert systems
A ``knowledge engineer'' interviews experts in a certain domain and tries to embody their knowledge in a computer program for carrying out some task. How well this works depends on whether the intellectual mechanisms required for the task are within the present state of AI. When this turned out not to be so, there were many disappointing results. One of the first expert systems was MYCIN in 1974, which diagnosed bacterial infections of the blood and suggested treatments. It did better than medical students or practicing doctors, provided its limitations were observed. Namely, its ontology included bacteria, symptoms, and treatments and did not include patients, doctors, hospitals, death, recovery, and events occurring in time. Its interactions depended on a single patient being considered. Since the experts consulted by the knowledge engineers knew about patients, doctors, death, recovery, etc., it is clear that the knowledge engineers forced what the experts told them into a predetermined framework. In the present state of AI, this has to be true. The usefulness of current expert systems depends on their users having common sense.
heuristic classification
One of the most feasible kinds of expert system given the present knowledge of AI is to put some information in one of a fixed set of categories using several sources of information. An example is advising whether to accept a proposed credit card purchase. Information is available about the owner of the credit card, his record of payment and also about the item he is buying and about the establishment from which he is buying it (e.g., about whether there have been previous credit card frauds at this establishment).

Branches of AI

2008-02-03T07:09:00.001-08:00

Q. What are the branches of AI?
A. Here's a list, but some branches are surely missing, because no-one has identified them yet. Some of these may be regarded as concepts or topics rather than full branches.
logical AI
What a program knows about the world in general the facts of the specific situation in which it must act, and its goals are all represented by sentences of some mathematical logical language. The program decides what to do by inferring that certain actions are appropriate for achieving its goals. The first article proposing this was [McC59]. [McC89] is a more recent summary. [McC96b] lists some of the concepts involved in logical aI. [Sha97] is an important text.
search
AI programs often examine large numbers of possibilities, e.g. moves in a chess game or inferences by a theorem proving program. Discoveries are continually made about how to do this more efficiently in various domains.
pattern recognition
When a program makes observations of some kind, it is often programmed to compare what it sees with a pattern. For example, a vision program may try to match a pattern of eyes and a nose in a scene in order to find a face. More complex patterns, e.g. in a natural language text, in a chess position, or in the history of some event are also studied. These more complex patterns require quite different methods than do the simple patterns that have been studied the most.
representation
Facts about the world have to be represented in some way. Usually languages of mathematical logic are used.
inference
From some facts, others can be inferred. Mathematical logical deduction is adequate for some purposes, but new methods of non-monotonic inference have been added to logic since the 1970s. The simplest kind of non-monotonic reasoning is default reasoning in which a conclusion is to be inferred by default, but the conclusion can be withdrawn if there is evidence to the contrary. For example, when we hear of a bird, we man infer that it can fly, but this conclusion can be reversed when we hear that it is a penguin. It is the possibility that a conclusion may have to be withdrawn that constitutes the non-monotonic character of the reasoning. Ordinary logical reasoning is monotonic in that the set of conclusions that can the drawn from a set of premises is a monotonic increasing function of the premises. Circumscription is another form of non-monotonic reasoning.
common sense knowledge and reasoning
This is the area in which AI is farthest from human-level, in spite of the fact that it has been an active research area since the 1950s. While there has been considerable progress, e.g. in developing systems of non-monotonic reasoning and theories of action, yet more new ideas are needed. The Cyc system contains a large but spotty collection of common sense facts.
learning from experience
Programs do that. The approaches to AI based on connectionism and neural nets specialize in that. There is also learning of laws expressed in logic. [Mit97] is a comprehensive undergraduate text on machine learning. Programs can only learn what facts or behaviors their formalisms can represent, and unfortunately learning systems are almost all based on very limited abilities to represent information.
planning
Planning programs start with general facts about the world (especially facts about the effects of actions), facts about the particular situation and a statement of a goal. From these, they generate a strategy for achieving the goal. In the most common cases, the strategy is just a sequence of actions.
epistemology
This is a study of the kinds of knowledge that are required for solving problems in the world.
ontology
Ontology is the study of the kinds of things that exist. In AI, the programs and sentences deal with various kinds of objects, and we study what these kinds are and what their basic properties are. Emphasis on ontology begins in the 1990s.
heuristics
A heuristic is a way of trying to discover something or an idea imbedded in a program. The term is used variously in AI. Heuristic functions are used in some approaches to search to measure how far a node in a search tree seems to be from a goal. Heuristic predicates that compare two nodes in a search tree to see if one is better than the other, i.e. constitutes an advance toward the goal, may be more useful. [My opinion].
genetic programming
Genetic programming is a technique for getting programs to solve a task by mating random Lisp programs and selecting fittest in millions of generations. It is being developed by John Koza's group and here's a tutorial.

Basic Questions about Artificial Intelligence

2008-02-03T07:06:00.000-08:00

Q. What is artificial intelligence?
A. It is the science and engineering of making intelligent machines, especially intelligent computer programs. It is related to the similar task of using computers to understand human intelligence, but AI does not have to confine itself to methods that are biologically observable.
Q. Yes, but what is intelligence?
A. Intelligence is the computational part of the ability to achieve goals in the world. Varying kinds and degrees of intelligence occur in people, many animals and some machines.
Q. Isn't there a solid definition of intelligence that doesn't depend on relating it to human intelligence?
A. Not yet. The problem is that we cannot yet characterize in general what kinds of computational procedures we want to call intelligent. We understand some of the mechanisms of intelligence and not others.
Q. Is intelligence a single thing so that one can ask a yes or no question ``Is this machine intelligent or not?''?
A. No. Intelligence involves mechanisms, and AI research has discovered how to make computers carry out some of them and not others. If doing a task requires only mechanisms that are well understood today, computer programs can give very impressive performances on these tasks. Such programs should be considered ``somewhat intelligent''.
Q. Isn't AI about simulating human intelligence?
A. Sometimes but not always or even usually. On the one hand, we can learn something about how to make machines solve problems by observing other people or just by observing our own methods. On the other hand, most work in AI involves studying the problems the world presents to intelligence rather than studying people or animals. AI researchers are free to use methods that are not observed in people or that involve much more computing than people can do.
Q. What about IQ? Do computer programs have IQs?
A. No. IQ is based on the rates at which intelligence develops in children. It is the ratio of the age at which a child normally makes a certain score to the child's age. The scale is extended to adults in a suitable way. IQ correlates well with various measures of success or failure in life, but making computers that can score high on IQ tests would be weakly correlated with their usefulness. For example, the ability of a child to repeat back a long sequence of digits correlates well with other intellectual abilities, perhaps because it measures how much information the child can compute with at once. However, ``digit span'' is trivial for even extremely limited computers.
However, some of the problems on IQ tests are useful challenges for AI.
Q. What about other comparisons between human and computer intelligence?
Arthur R. Jensen [Jen98], a leading researcher in human intelligence, suggests ``as a heuristic hypothesis'' that all normal humans have the same intellectual mechanisms and that differences in intelligence are related to ``quantitative biochemical and physiological conditions''. I see them as speed, short term memory, and the ability to form accurate and retrievable long term memories.
Whether or not Jensen is right about human intelligence, the situation in AI today is the reverse.
Computer programs have plenty of speed and memory but their abilities correspond to the intellectual mechanisms that program designers understand well enough to put in programs. Some abilities that children normally don't develop till they are teenagers may be in, and some abilities possessed by two year olds are still out. The matter is further complicated by the fact that the cognitive sciences still have not succeeded in determining exactly what the human abilities are. Very likely the organization of the intellectual mechanisms for AI can usefully be different from that in people.
Whenever people do better than computers on some task or computers use a lot of computation to do as well as people, this demonstrates that the program designers lack understanding of the intellectual mechanisms required to do the task efficiently.
Q. When did AI research start?
A. After WWII, a number of people independently started to work on intelligent machines. The English mathematician Alan Turing may have been the first. He gave a lecture on it in 1947. He also may have been the first to decide that AI was best researched by programming computers rather than by building machines. By the late 1950s, there were many researchers on AI, and most of them were basing their work on programming computers.
Q. Does AI aim to put the human mind into the computer?
A. Some researchers say they have that objective, but maybe they are using the phrase metaphorically. The human mind has a lot of peculiarities, and I'm not sure anyone is serious about imitating all of them.
Q. What is the Turing test?
A. Alan Turing's 1950 article Computing Machinery and Intelligence [Tur50] discussed conditions for considering a machine to be intelligent. He argued that if the machine could successfully pretend to be human to a knowledgeable observer then you certainly should consider it intelligent. This test would satisfy most people but not all philosophers. The observer could interact with the machine and a human by teletype (to avoid requiring that the machine imitate the appearance or voice of the person), and the human would try to persuade the observer that it was human and the machine would try to fool the observer.
The Turing test is a one-sided test. A machine that passes the test should certainly be considered intelligent, but a machine could still be considered intelligent without knowing enough about humans to imitate a human.
Daniel Dennett's book Brainchildren [Den98] has an excellent discussion of the Turing test and the various partial Turing tests that have been implemented, i.e. with restrictions on the observer's knowledge of AI and the subject matter of questioning. It turns out that some people are easily led into believing that a rather dumb program is intelligent.
Q. Does AI aim at human-level intelligence?
A. Yes. The ultimate effort is to make computer programs that can solve problems and achieve goals in the world as well as humans. However, many people involved in particular research areas are much less ambitious.
Q. How far is AI from reaching human-level intelligence? When will it happen?
A. A few people think that human-level intelligence can be achieved by writing large numbers of programs of the kind people are now writing and assembling vast knowledge bases of facts in the languages now used for expressing knowledge.
However, most AI researchers believe that new fundamental ideas are required, and therefore it cannot be predicted when human-level intelligence will be achieved.
Q. Are computers the right kind of machine to be made intelligent?
A. Computers can be programmed to simulate any kind of machine.
Many researchers invented non-computer machines, hoping that they would be intelligent in different ways than the computer programs could be. However, they usually simulate their invented machines on a computer and come to doubt that the new machine is worth building. Because many billions of dollars that have been spent in making computers faster and faster, another kind of machine would have to be very fast to perform better than a program on a computer simulating the machine.
Q. Are computers fast enough to be intelligent?
A. Some people think much faster computers are required as well as new ideas. My own opinion is that the computers of 30 years ago were fast enough if only we knew how to program them. Of course, quite apart from the ambitions of AI researchers, computers will keep getting faster.
Q. What about parallel machines?
A. Machines with many processors are much faster than single processors can be. Parallelism itself presents no advantages, and parallel machines are somewhat awkward to program. When extreme speed is required, it is necessary to face this awkwardness.
Q. What about making a ``child machine'' that could improve by reading and by learning from experience?
A. This idea has been proposed many times, starting in the 1940s. Eventually, it will be made to work. However, AI programs haven't yet reached the level of being able to learn much of what a child learns from physical experience. Nor do present programs understand language well enough to learn much by reading.
Q. Might an AI system be able to bootstrap itself to higher and higher level intelligence by thinking about AI?
A. I think yes, but we aren't yet at a level of AI at which this process can begin.
Q. What about chess?
A. Alexander Kronrod, a Russian AI researcher, said ``Chess is the Drosophila of AI.'' He was making an analogy with geneticists' use of that fruit fly to study inheritance. Playing chess requires certain intellectual mechanisms and not others. Chess programs now play at grandmaster level, but they do it with limited intellectual mechanisms compared to those used by a human chess player, substituting large amounts of computation for understanding. Once we understand these mechanisms better, we can build human-level chess programs that do far less computation than do present programs.
Unfortunately, the competitive and commercial aspects of making computers play chess have taken precedence over using chess as a scientific domain. It is as if the geneticists after 1910 had organized fruit fly races and concentrated their efforts on breeding fruit flies that could win these races.
Q. What about Go?
A. The Chinese and Japanese game of Go is also a board game in which the players take turns moving. Go exposes the weakness of our present understanding of the intellectual mechanisms involved in human game playing. Go programs are very bad players, in spite of considerable effort (not as much as for chess). The problem seems to be that a position in Go has to be divided mentally into a collection of subpositions which are first analyzed separately followed by an analysis of their interaction. Humans use this in chess also, but chess programs consider the position as a whole. Chess programs compensate for the lack of this intellectual mechanism by doing thousands or, in the case of Deep Blue, many millions of times as much computation.
Sooner or later, AI research will overcome this scandalous weakness.
Q. Don't some people say that AI is a bad idea?
A. The philosopher John Searle says that the idea of a non-biological machine being intelligent is incoherent. He proposes the Chinese room argument www-formal.stanford.edu/jmc/chinese.html The philosopher Hubert Dreyfus says that AI is impossible. The computer scientist Joseph Weizenbaum says the idea is obscene, anti-human and immoral. Various people have said that since artificial intelligence hasn't reached human level by now, it must be impossible. Still other people are disappointed that companies they invested in went bankrupt.
Q. Aren't computability theory and computational complexity the keys to AI? [Note to the layman and beginners in computer science: These are quite technical branches of mathematical logic and computer science, and the answer to the question has to be somewhat technical.]
A. No. These theories are relevant but don't address the fundamental problems of AI.
In the 1930s mathematical logicians, especially Kurt Gödel and Alan Turing, established that there did not exist algorithms that were guaranteed to solve all problems in certain important mathematical domains. Whether a sentence of first order logic is a theorem is one example, and whether a polynomial equations in several variables has integer solutions is another. Humans solve problems in these domains all the time, and this has been offered as an argument (usually with some decorations) that computers are intrinsically incapable of doing what people do. Roger Penrose claims this. However, people can't guarantee to solve arbitrary problems in these domains either. See my Review of The Emperor's New Mind by Roger Penrose. More essays and reviews defending AI research are in [McC96a].
In the 1960s computer scientists, especially Steve Cook and Richard Karp developed the theory of NP-complete problem domains. Problems in these domains are solvable, but seem to take time exponential in the size of the problem. Which sentences of propositional calculus are satisfiable is a basic example of an NP-complete problem domain. Humans often solve problems in NP-complete domains in times much shorter than is guaranteed by the general algorithms, but can't solve them quickly in general.
What is important for AI is to have algorithms as capable as people at solving problems. The identification of subdomains for which good algorithms exist is important, but a lot of AI problem solvers are not associated with readily identified subdomains.
The theory of the difficulty of general classes of problems is called computational complexity. So far this theory hasn't interacted with AI as much as might have been hoped. Success in problem solving by humans and by AI programs seems to rely on properties of problems and problem solving methods that the neither the complexity researchers nor the AI community have been able to identify precisely.
Algorithmic complexity theory as developed by Solomonoff, Kolmogorov and Chaitin (independently of one another) is also relevant. It defines the complexity of a symbolic object as the length of the shortest program that will generate it. Proving that a candidate program is the shortest or close to the shortest is an unsolvable problem, but representing objects by short programs that generate them should sometimes be illuminating even when you can't prove that the program is the shortest.

eDirectory Authentication using LdapConnection and custom certificate validation

2008-01-11T08:34:00.000-08:00

Introduction
This article explains how to authenticate a user over LDAPS using the System.DirectoryServices.Protocols.LdapConnection class, performing custom certificate validation.

Background
Recently, I ran into trouble using System.DirectoryServices.DirectoryEntry to connect to a Novell eDirectory server because the certificate was self-signed. When run in an ASP.NET application, the machine-level certificate store was not examined. So, even though the self-signed certificate was in the trusted store, DirectoryEntry was still refusing to establish a connection.

The LdapConnection class is a better choice for this situation, as it allows the user to validate the certificate manually. Note that the DirectoryEntry approach works fine with a trusted self-signed certificate when run in a Windows Forms application. A good example of using DirectoryEntry to connect to eDirectory can be found here.

Using the code
The example uses a Login control for simplicity. I recommend that in the "real world" you create a custom MembershipProvider. The following example code was used to connect to a Novell eDirectory server over secure LDAP. However, the code should work fine with other directory providers as long as the correct server/port/root DN is used.

Also remember to omit the con.SessionOptions.SecureSocketLayer = true line if you aren't using LDAPS. However, if you aren't using LDAPS you might as well use DirectoryEntry!

Connecting and authenticating
First, we set up our LdapConnection class. We specify the address and port of the server that we will be connecting over SSL, set up the certificate callback (more on that later) and provide the default credentials for authentication.

This example uses eDirectory's "contextless login" feature, so blank credentials are allowed. Depending on your LDAP server, you may need to specify credentials to search the directory. Also note that if no port is specified, then the default value of 389 will be used.

LdapConnection con = new LdapConnection(new LdapDirectoryIdentifier(
"EDIRECTORYSERVER:636"));
con.SessionOptions.SecureSocketLayer = true;
con.SessionOptions.VerifyServerCertificate =
new VerifyServerCertificateCallback(ServerCallback);
con.Credential = new NetworkCredential(String.Empty, String.Empty);
con.AuthType = AuthType.Basic;

Now we bind the initial connection. If con.Bind() executes without throwing an exception, the server and credentials specified are valid.

using (con)
{
con.Bind();

The next step is to search for the user's fully qualified, distinguished name. This is a necessary step because when users provide their usernames, they don't provide the full context of their names in the directory; i.e. jlennon is provided instead of cn=jlennon,ou=Beatles,ou=Artists,o=AppleRecordsLDAPDirectory.

First, we prepare the SearchRequest object. We specify the root DN, the search filter and the search scope. Then we send the request.

SearchRequest request = new SearchRequest(
"o=LDAPRoot",
"(&(objectClass=Person)(uid=" + Login1.UserName + "))",
SearchScope.Subtree);

SearchResponse response = (SearchResponse)con.SendRequest(request);

If we get this far without an exception being thrown, we know that the root DN and search filter specified are valid. If either is invalid, a DirectoryOperationException will be thrown.

Now we can extract the DN from the search result. If you want to provide a "no such username" message, you can check that response.Entries.Count > 0. An ArgumentOutOfRangeException will be thrown on the following line if the Username provided does not exist.

SearchResultEntry entry = response.Entries[0];
string dn = entry.DistinguishedName;

Now that we have the full DN for the user, we can check if the given password is valid. We set a new NetworkConnection on the LdapConnection object and re-bind. con.Bind() will throw LdapException if the password provided is invalid.

con.Credential = new NetworkCredential(dn, Login1.Password);
con.Bind();
}

If we get this far, we have successfully authenticated! We can now use a SearchRequest to search for group membership, etc. However, that's out of scope for this example.

VerifyServerCertificateCallback
This should need no explanation. We simply load the certificate file from disk and compare it to the certificate presented by the server. Production code should handle exceptions associated with reading the certificate: File not found, access denied, etc. If you really trust the server, you could omit all of this and just return true. ;)

public static bool ServerCallback(
LdapConnection connection, X509Certificate certificate)
{
try
{
X509Certificate expectedCert =
X509Certificate.CreateFromCertFile(
"C:\\certificates\\certificate.cer");

if (expectedCert.Equals(certificate))
{
return true;
}
else
{
return false;
}
}
catch (Exception ex)
{
return false;
}
}

Source Code Line Counter

2008-01-11T08:33:00.000-08:00

Introduction
While updating my cv, I wanted a quick way of seeing how many lines of code my application contained (160,430 lines!). I haven't looked into Visual Studio Add-ins too much so I wrote a simple stand-alone application that has a few nice features.

Using the Application
Fire up the application and select a folder which contains your code or projects - that's it. The application then searches for all code files and counts every line of code. It displays the files included along the line count. It then generates a total at the bottom.

Checking the "Include subfolders" will instruct the software to search all the sub folders, and you also have the option to look for particular source files, *.cs for C# and *.vb for VB.net.

Nice Feature
Some of my projects are spread across two folders, (client / server) so I needed a way of adding these two totals together. After calculating the total for one folder, you can click on the Grouping > Add menu to add the total and directory to a list. As you tally up the folders a total is show in the menu.

Not a lot of Code
There isn't a lot of code to this but here's the important parts.

Using the DirectoryInfo.GetFiles function allows for easy use of searching just a folder or all folders contained within it with a search pattern, *.cs or *.vb. It's fast and saves time having to write your own recursive function.

DirectoryInfo dir = new DirectoryInfo(folderBrowserDialog1.SelectedPath);
FileInfo[] files = dir.GetFiles(cmbSearchPattern.SelectedItem.ToString(),
(chkIncludeSubfolders.Checked ? SearchOption.AllDirectories :
SearchOption.TopDirectoryOnly));

In case you aren't familiar with the inline if statement I hope this makes it a little clearer.

(true_or_false ? Do_this_if_true : Do_this_if_false)
(chkIncludeSubfolders.Checked ? SearchOption.AllDirectories :
SearchOption.TopDirectoryOnly));

Counting the lines
I figured that to count the lines I should open the text and use a RegEx "\n" to calculate the lines, but I guess that the RegEx is doing the same thing I would which is open the file and read each line. So I just use a loop to count the lines, which is just as quick I reckon.

reader = file.OpenText();
while (reader.ReadLine() != null)
{
fileCounter++;
}
reader.Close();

Collection.RemoveAt
Got stuck trying to remove items within a collection because I only need to remove a certain amount of items from a particular index. Items 0 to 3 contain the Add, Clear All, Total and a Separator. This solution below sorts this out.

// if there was two items or more in the collection then
// one item would always be left.
for (int i = 4; i < toolStripGrouping.DropDownItems.Count; i++)
{
toolStripGrouping.DropDownItems.RemoveAt(i);
}
// this fixes it
int count = toolStripGrouping.DropDownItems.Count;
for (int i = 4; i < count; i++)
{
toolStripGrouping.DropDownItems.RemoveAt(4);
}

Line Counter Code
The updated line counter code was changed to include or exclude certain lines. The easiest way for me to do this was to first remove the "\t" tabs from the line and then investigate the line of code.

Collapse // Replace the tabs
s = s.Replace("\t", "");

// Comment
if (s.StartsWith("//"))
{
dontCount = true;
}

// Blank line
if (s == "")
{
dontCount = true;
}

// This autoGenerateCode flag lets the software know that
// it should be looking for the first #endregion to close
// the starting one.
if (autoGeneratedCode)
{
if (s.Contains("#endregion"))
{
dontCount = true;
autoGeneratedCode = false;
}
}
else if (s == "#region Windows Form Designer generated code")
{
dontCount = true;
autoGeneratedCode = true;
}

History of Searches
I thought that re-browsing projects I have already counted was getting a bit repetitive so adding a history made sense.

Basically it works like this: if a search is performed it checks to see if it exists in the history stored in the ToolStripItems. If it does then it just moves that ToolStripItem to the top using the Insert function with a index set to 0. If it doesn't then it first checks to see if it has reached the maximum amount allowed (currently 5), removing the last one if it has and inserting the new search into index.

Collapseprivate void AddToFolderHistory(string FolderPath)
{
// Check if folder exists in the list
ToolStripItem[] searchItems = btnBrowse.DropDownItems.Find(FolderPath,
true);
if (searchItems.Length == 0)
{
// Doesn't exist
if (btnBrowse.DropDownItems.Count == maximumHistory)
{
btnBrowse.DropDownItems.RemoveAt(maximumHistory-1);
}

ToolStripMenuItem item = new ToolStripMenuItem();
item.Name = item.Text = FolderPath;
item.Click += new EventHandler(FolderHistory_Click);
this.btnBrowse.DropDownItems.Insert(0, item);
}
else
{
// Does exist so just move to the top
this.btnBrowse.DropDownItems.Insert(0, searchItems[0]);
}
}
private void FolderHistory_Click(object sender, EventArgs e)
{
// History item has been selected so move to the top
lblFolder.Text = ((ToolStripMenuItem)sender).Text;
this.btnBrowse.DropDownItems.Insert(0, ((ToolStripMenuItem)sender));
CountFiles(lblFolder.Text);
}

Conclusion
In my opinion, a neat little application that does exactly what it says on the tin. It's helped with my cv but no job offers just yet!

History
Added the ability to not count blank lines, comments, Visual Studio auto-generated code, or designer files. This is accessed through the File Options menu items.

Changed the search routine so that you can browse multiple extensions (*.cs|*.vb) at the same time.
Copy the current list of files and number of lines to the clipboard for email/printing
Redesigned the Browse feature with a Toolbar
Added a Recount button
A quick history list of the last 5 (configurable in code) searches - explained below.
Add the additional code to filter vb files.

If someone could send me the "#region" and "#endregion" in C++.net I can add this to the routine as well.

No ASPX just yet. I have a real software job to do.

License

Docking Toolbars in Plain C

2008-01-11T08:31:00.000-08:00

First, we'll discuss the "floating" aspect of tool windows - i.e. how to get tool windows to stay floating on top of all other windows, how to get the window activation working correctly, etc. Later, we'll see how to get these floating tool windows to "dock" to a side of the owner window, and discuss various methods of window management.

The source code presented will culminate in a Win32 DLL called DockWnd.dll whose functions can be used by any Win32 program to easily support docking tool windows. There are many different ways to create a docking window. I imagine most of the code on the Internet uses well-designed C++ classes to hide the implementation. However, I still prefer to code in C, so the design of this library will be a non-object oriented approach and therefore easily useable by an application written in any language.

This code/article is based upon some original free code provided by James Brown. His website, contains an earlier, different version of this code (as well as numerous other free Win32 tutorials/examples).

Contents
DOCKINFO structure
How DockingCreateFrame creates a floating tool window
Prevent tool window deactivation
Show all tool windows as active
Sync the activation of all tool windows
Sync the enabled state of all tool windows
Docking a tool window
Floating versus docked size
Moving a window with a drag-rectangle
Drawing a drag-rectangle
Redrawing the docked windows
Enumerating tool windows
Various other features of the library
An application
Creating a tool window
Handling WM_SIZE message in the owner
Handling WM_NCACTIVATE message in the owner
Handling WM_ENABLE message in the owner
Handling messages sent to standard controls
Multiple child windows inside a tool window
Closing a tool window
Saving/Restoring a tool window size/position
Miscellaneous remarks
DOCKINFO structure
Because our docking library needs to maintain some information about each tool window it manages, we need some structure to store the information. We'll define a DOCKINFO structure (in dockwnd.h), and allocate a DOCKINFO for each tool window created. The application will call the DockingAlloc function in our docking library to allocate a DOCKINFO (initialized to default values), and then pass it to the DockingCreateFrame function which creates the actual tool window associated with that DOCKINFO. This struct will hold information such as the handle (HWND) to the tool window, the handle to the tool window's owner window, whether the tool window is currently docked to its owner or floating, and other information for our private use in managing docking tool windows. We store the handle to the tool window in the DOCKINFO's hwnd field. We store the handle to the owner window in the DOCKINFO's container field. And the value of the DOCKINFO's uDockedState field tells whether the tool window is floating or docked. This field is OR'ed with DWS_FLOATING (and therefore a negative value) when a tool window is floating. We'll discuss the other DOCKINFO fields later.

Note: It is the application's responsibility to create and manage the owner window. Our library deals only with creating/managing the tool windows.

We register our own window class (with the class name of "DockWnd32") for our tool windows. The window procedure (dockWndProc) for this class is inside of our library. We use the GWL_USERDATA field of the tool window to store a pointer to that tool window's DOCKINFO struct. In this way, we can easily and quickly fetch the appropriate DOCKINFO given only a handle (HWND) to a particular tool window.

We don't want to limit the application to only one owner window, and its set of docked windows. For example, perhaps an application will have two owner windows, each with its own set of docked windows. Nor, do we want to limit the application to a particular number of tool windows. So, we may be asked to create DOCKINFOs for numerous sets of tool windows and owners. By storing a pointer to a tool window's DOCKINFO in the tool window itself, and storing handles to the tool window and its owner window inside the DOCKINFO, we can easily get all of the information we need for our library to do what it needs to do, with minimal work on the part of the application.

There are times when our library needs to be able to enumerate all tool windows for a particular owner window. We'll get to the particulars of that later. In some of our example code below, we'll just refer to a placeholder function called DockingNextToolWindow which you should assume will fetch the next tool window (actually, that tool window's DOCKINFO) for a particular owner window. In the actual source code for the library, this is replaced by more complex code that we'll examine later.

How DockingCreateFrame creates a floating tool window
A floating tool window is just a standard window with the WS_POPUP style. When a popup window is created with an owner window, the popup is positioned so that it always stays on top of that owner window. This is how we can create and display a floating tool window:

// Create a floating (popup) tool window
HWND hwnd = CreateWindowEx(
WS_EX_TOOLWINDOW,
"ToolWindowClass", "ToolWindow",
WS_POPUP | WS_SYSMENU | WS_THICKFRAME | WS_CAPTION | WS_VISIBLE,
200, 200, 400, 64,
hwndOwner, NULL, GetModuleHandle(0), NULL
);

Note: In the above example, it is assumed that hwndOwner is a handle to some other window the application created to be the owner window for our tool window. The application must create this window, and then pass its handle to DockingCreateFrame. In other words, the application must create the owner window before any tool window can be created for it.

The WS_EX_TOOLWINDOW extra style doesn't do anything special, other than to make a window with a smaller titlebar. It doesn't make the window magically float - this is achieved automatically by specifying WS_POPUP style and an owner window. Here's what the above CreateWindowEx may display:

A tool window floating above its owner window ("Main window").

Prevent window deactivation
The image above shows the owner window ("Main window") with an inactive titlebar. This is entirely normal, because only one window at a time can have the input focus, and the operating system normally shows only that one window with an active titlebar. So, when we create our tool window, the operating system shows the tool window as active, and shows the owner window as deactivated.

But, it is normal practice for tool windows and their owner window to appear active at the same time. It looks more natural this way. So we need to devise a strategy to keep our tool window and owner window both appear active, even if only one technically has the input focus.

The solution involves the WM_NCACTIVATE message. The operating system sends this message when a window's non-client area (the titlebar and border) needs to be activated or deactivated. As with all window messages, WM_NCACTIVATE is sent with two parameters - wParam and lParam. When a window receives WM_NCACTIVATE with wParam=TRUE, this indicates that the titlebar and border should be shown as active. When wParam=FALSE, this indicates that the titlebar and border should be shown as inactive.

Note: MSDN states that WM_NCACTIVATE's lParam will always be 0. However, I have observed that lParam indicates the window handle of the window being deactivated. This appears to be true under Win95, 98 and NT, 2000, XP. Our solution relies upon this undocumented feature.

So, when we create our tool window, our owner window receives a WM_NCACTIVATE with wParam=FALSE and our tool window receives a WM_NCACTIVATE with wParam=TRUE.

When this message is passed to DefWindowProc(), the operating system does two things. First, the titlebar is drawn as either active or inactive, depending upon whether wParam is TRUE or FALSE respectively. Secondly, the operating system sets an internal flag for the window which remembers if the window was painted as active or inactive. This enables DefWindowProc() to process subsequent WM_NCPAINT messages to paint the titlebar with the proper activation. It is advisable to always pass WM_NCACTIVATE to DefWindowProc() so that this internal flag is set, even if you also do your own processing of WM_NCACTIVATE.

This WM_NCACTIVATE message provides us with a way to make all our tool windows, and owner window, look active, even if only one window technically has the focus. To do this, whenever our tool windows or owner window receive a WM_NCACTIVATE, we will always substitute TRUE for wParam when we pass the WM_NCACTIVATE to DefWindowProc(). The result is that the operating system always renders the titlebars of our tool windows and owner window as active.

Here is some code we could add to the window procedure of all our tool windows, and owner window, to show them all as simultaneously active:

case WM_NCACTIVATE:
return DefWindowProc(hwnd, msg, TRUE, lParam);

Both the tool window and its owner window are shown active.

Incidentally, MDI child windows also use this same technique to keep their titlebars active. The only difference is that MDI windows have the WS_CHILD style, instead of WS_POPUP.

Show all tool windows as active
The above method seems to do what we want, but there is a problem. Our owner window and tool windows will always appear active, even if our application is not in the foreground. For example, if the end user switches to some other application's window, our tool window and owner window still will look active, which can be a bit disconcerting to the end user.

Also, whenever we display a message box or a normal dialog box, the owner window and tool windows will still appear active, when in this scenario we ideally want to make them look inactive.

This calls for a more careful study of window activation messages. Here is a description of the series of window activation messages sent when one window becomes active, and another inactive:

WM_MOUSEACTIVATE is sent to the window about to become active, to ask it whether or not the activation request should be allowed. What your window procedure returns (i.e., MA_ACTIVATE or MA_NOACTIVATE) affects the subsequent activation messages.
WM_ACTIVATEAPP is sent if a window belonging to a different application is about to become active (or inactive). This message is sent both to the window that is currently active (to tell it that it is about to become inactive) as well as the window that is about to become active. The return value should always be zero, and never affects subsequent messages' behaviour.
As described above, WM_NCACTIVATE is sent when a window's non-client area needs to be shown activated or deactivated.
WM_ACTIVATE is sent last of all to the window becoming active. When this message is passed to DefWindowProc(), the operating system sets the input focus to that window.
With all of these activation messages, only two windows are actually involved - the window being deactivated, and the window being activated. So, even if we have many floating tool windows, not all of them will receive these messages. Only the one window being activated, and the one window being deactivated, receive the messages. But to make things look and feel right, we want the displayed state (i.e., whether a tool window's titlebar is shown active or inactive) of each tool window to be the same as all other tool windows. So, even though not all of our windows will receive the above messages, we still need to have all windows synced to the same state.

This same discussion applies whenever we need to disable or enable our owner window. If the owner window is to be disabled or enabled, we want to sync all the tool windows to that same state. (But, a different set of messages are sent for a window being disabled or enabled.)

So, our docking library has a bit of work to do in order to make things look and feel right:

When our application is activated / deactivated, we need to sync the active / inactive display of all tool windows with each other.
This also applies to activation within our own application. For example, if the user activates a window we create that isn't one of our tool windows, then we want to show all tool windows deactivated. And if the user switches back to a tool window from that window, we want all tool windows shown active.

When the owner window is disabled due to a modal dialog or message box being displayed, then we must disable all tool windows (and any modeless dialogs) to prevent the user from interacting with them while the modal dialog / message box is on screen.
A first try at a solution
Our first stab at a solution will be to concentrate on the WM_ACTIVATE message. This message is received whenever a window is activated or deactivated. The direction we will take will be to decide if the window receiving this message is active or inactive, and synchronise all other windows to the same state by manually sending them a "spoof" WM_NCACTIVATE message. This spoof message will force the other windows to update their titlebars to the same state as the window receiving the WM_ACTIVATE.

Here's a function that we could add to our docking library. Whenever one of our tool windows, or owner window, receives a WM_ACTIVATE message, it will call this function to sync the state of all tool windows:

Collapse/*********************** DockingActivate() **********************
* Sends WM_NCACTIVATE to all the owner's tool windows. A
* tool or owner window calls this in response to receiving
* a WM_ACTIVATE message.
*
* container = Handle to owner window.
* hwnd = Handle to window which received WM_ACTIVATE (can
* be the owner, or one of its tool windows).
* wParam = WPARAM of the WM_ACTIVATE message.
* lParam = LPARAM of the WM_ACTIVATE message.
*/

LRESULT WINAPI DockingActivate(HWND container,
HWND hwnd, WPARAM wParam, LPARAM lParam)
{
DOCKINFO * dwp;
BOOL fKeepActive;

fKeepActive = (wParam != WA_INACTIVE);

// Get the DOCKINFO of the next tool window for this owner window. when 0
// is returned, there are no more tool windows for this owner.
while ((dwp = DockingNextToolWindow(container)))
{
// Sync this tool window to the same state as the window that called
// DockingActivate.
SendMessage(dwp->hwnd, WM_NCACTIVATE, fKeepActive, 0);
}

// Allow the window that called DockingActivate to handle its WM_NCACTIVATE
// as normally it would.
return DefWindowProc(hwnd, WM_ACTIVATE, wParam, lParam);
}

It works, after a fashion. All tool windows activate and deactivate correctly, and all at the same time. This solution is not the best though.

The problem is that every tool window's titlebar flashes whenever the active window changes. This is because of the way the operating system sends the WM_ACTIVATE message. This message is first sent to the window that is being deactivated. If that happens to be a tool window or the owner window, it will call DockingActivate to deactivate all the tool windows. WM_ACTIVATE is then sent to the active window. If that window also happens to be a tool window or owner window, it will call DockingActivate to (correctly) activate all the tool windows. It is the fact that DockingActivate is quickly called twice (once to deactivate the tool windows, and then to activate them) that causes all the windows to flash.

A partial solution is to perform a check before deactivating the tool windows. We know that if a window is being deactivated, lParam identifies the (other) window about to be activated. And if this other window is one of our tool windows, we can skip deactivating the tool windows, because we know the other (tool) window is going to subsequently activate them anyway.

if (fKeepActive == FALSE)
{
while ((dwp = DockingNextToolWindow(container)))
{
if (dwp->hwnd == (HWND)lParam)
return DefWindowProc(hwnd, WM_ACTIVATE, wParam, lParam);
}
}

This prevents every tool window from briefly deactivating, then activating again. There is still a problem, albeit a minor one. The problem is, the single tool window that is being deactivated will still flicker briefly before being activated again. This is because it will already have received its WM_NCACTIVATE message, which caused the window to be redrawn deactivated. The window gets its activated look eventually, but this brief flicker is still visible.

Sync the activation of all tool windows
We need to take a step back and approach the problem from a slightly different direction. Instead of handling WM_ACTIVATE, which is called after a window's titlebar is redrawn, we'll go straight to the heart of the problem, and rewrite DockingActivate to be called whenever a window receives a WM_NCACTIVATE message. This will ensure that no unnecessary activation or deactivation will take place.

The function presented below performs several tasks on behalf of the tool (or owner) window that calls DockingActivate:

Search the list for the other window being activated/deactivated (the window specified by lParam, rather than the window receiving WM_NCACTIVATE). If this other window is a tool window, then we force all tool windows as activated.
Synchronize all current tool windows to our (possibly new) state.
Activate/deactivate the window that calls DockingActivate, depending on our new state.
The code looks like this:

CollapseLRESULT WINAPI DockingActivate(HWND container,
HWND hwnd, WPARAM wParam, LPARAM lParam)
{
DOCKINFO * dwp;
BOOL fKeepActive;
BOOL fSyncOthers;

// If this is a spoof'ed message we sent, then handle it
// normally (but reset LPARAM to 0).
if (lParam == -1)
return DefWindowProc(hwnd, WM_NCACTIVATE, wParam, 0);

fKeepActive = wParam;
fSyncOthers = TRUE;

while ((dwp = DockingNextToolWindow(container)))
{
// UNDOCUMENTED FEATURE:
// If the other window being activated/deactivated (i.e. not the one that
// called here) is one of our tool windows, then go (or stay) active.
if ((HWND)lParam == dwp->hwnd)
{
fKeepActive = TRUE;
fSyncOthers = FALSE;
break;
}
}

if (fSyncOthers == TRUE)
{
// Sync all other tool windows to the same state.
while ((dwp = DockingNextToolWindow(container)))
{
// Send a spoof'ed WM_NCACTIVATE message to this tool window,
// but not if it is the same window that called here. Note that
// we substitute a -1 for LPARAM to indicate that this is a
// spoof'ed message we sent. The operating system would never
// send a WM_NCACTIVATE with LPARAM = -1.
if (dwp->hwnd != hwnd && hwnd != (HWND)lParam)
SendMessage(dwp->hwnd, WM_NCACTIVATE, fKeepActive, -1);
}
}

return DefWindowProc(hwnd, WM_NCACTIVATE, fKeepActive, lParam);
}

The code above uses an undocumented feature of the WM_NCACTIVATE message which I observed while experimenting with these activation messages. The MSDN documentation states that lParam is unused (presumably zero), but this is not the case under Windows 95, 98, ME, and NT, 2000, XP.

Instead, lParam is a handle to the other window being activated/deactivated in our place (i.e., if we are being deactivated, lParam will be the handle to the window being activated). This is not always the case, specifically when the other window being activated/deactivated belongs to another process. In this case, lParam will be zero.

Sync the enabled state of all tool windows
Now, we need to tackle the other problem. When our owner window is disabled (perhaps because a modal dialog or message box has popped up), we need to disable all tool windows too. This feature prevents the user from clicking on and activating not only the main window, but also any tool window, while the modal dialog or message box is displayed.

The solution is similar to how we solved the activation problem, except that this time we write a function that a tool window or the owner window calls whenever it receives a WM_ENABLE message. DockingEnable simply enables/disables all the tool windows to the same state as the owner window.

Collapse/*********************** DockingEnable() **********************
* Sends WM_ENABLE to all the owner's tool windows.
* A window calls this in response to receiving a
* WM_ENABLE message.
*
* container = Handle to owner window.
* hwnd = Handle to window which received WM_ENABLE (can
* be the owner, or one of its tool windows).
* wParam = WPARAM of the WM_ENABLE message.
* lParam = LPARAM of the WM_ENABLE message.
*/

LRESULT WINAPI DockingEnable(HWND container,
HWND hwnd, WPARAM wParam, >LPARAM lParam)
{
DOCKINFO * dwp;

while ((dwp = DockingNextToolWindow(container)))
{
// Sync this tool window to the same state as the window that called
// DockingEnable (but not if it IS the window that called here).
if (dwp->hwnd != hwnd) EnableWindow(dwp->hwnd, wParam);
}

// Allow the window that called DockingEnable to handle its WM_ENABLE
// as normally it would.
return DefWindowProc(hwnd, WM_ENABLE, wParam, lParam);
}

Docking a tool window
The previous discussion took you through the steps necessary to create floating tool windows. Now, we'll discuss the techniques necessary to get these floating windows to "dock" with their owner window. I'm not going to reproduce all the library's source code in this tutorial, because quite a lot is involved. I'm instead going to give an overview of the approach taken, and you can study the profusely commented source code for details.

First of all, we need to define the terms "Docked" and "Undocked". A tool window is undocked when it is floating. And as we already know, in order to make that happen, the tool window must have the WS_POPUP style.

On the other hand, a tool window is docked when it is visually contained completely within its owner window, alongside one of the owner's borders. In order to make this happen, we must create the tool window with the WS_CHILD (not WS_POPUP) style (or change the style from WS_POPUP to WS_CHILD), and make its owner window also its parent window. When a tool window has the WS_CHILD style, the operating system restricts it to the area inside of its parent window, and the tool window is graphically "anchored" to its parent window (i.e., when the end user moves the parent window, the child window automatically moves with it).

But note that when the parent window is resized, the parent window will need to also move/resize the docked tool window so that the tool window remains "attached" to the border. (Of course, our library has functions the owner window can call to make this as easy as possible.)

A good docking library must allow the end user to be able to dock and undock any tool window by grabbing the tool window with the mouse and dragging it over to a dockable or undockable area. There are many different ways to implement docking windows. This is because there is no standard, built-in docking window support in Windows. Application developers have had to implement their own docking windows, or rely upon third party libraries to do the work for them (such as MFC).

There are two common types of docking window implementations. The most common (and intuitive, in my opinion) is the type where you grab the tool window (by a "gripper bar" or its title bar) with the mouse, and drag it around the screen. When you drag the tool window, instead of the window itself moving, a drag-rectangle (feedback rectangle) is XORed on the screen, showing the outline of where the window will move to when you release the mouse - like the way windows work when full-window dragging is turned off. With this method, when a window is dragged to / from a window, the feedback rectangle visibly changes to indicate that the window can be dropped. This is the docking implementation that our docking library uses.

A tool window being dragged. You can see the drag rectangle.

The second type of docking implementation can be found in some newer style applications (such as Microsoft Outlook). Instead of a feedback rectangle, windows can be directly "teared" or "snapped" on or off the owner window - i.e., they snap into place as soon as you manipulate them. Personally, I don't like this type of user-interface, and our docking library does not use it.

Our tool windows will have the following characteristics:

A docked tool window will have a "gripper bar" along its left side to allow the user to grab it and undock it.
A tool window will use a feedback (drag) rectangle as it's moved around the screen - even if the "full window drag" system setting is in effect. This is shown in the picture above.
While a drag-rectangle is dragged around the screen, at some point it will intersect one of the borders of its owner window. When this happens, the drag-rectangle will need to visibly change in order to reflect the fact that the tool window is now within a docking "region". Normal convention is for a wide (say three pixel) shaded rectangle to represent a floating position, and for a single-pixel rectangle to represent a docked position.
When the mouse is released after dragging a tool window, a test must be made to see if the window should be made to dock or float. (i.e., was the drag-rectangle ultimately moved to one of these docking "regions", or is it outside of any such region and therefore the tool window is floating?)
At the end user's discretion, a tool window can be forced to float, even when the drag-rectangle is released over a dockable area. This is usually achieved by the end user holding the key down.
When floating, a tool window can be resized just like any normal window. No special processing is required to do this - the standard Windows sizing behaviour can be used in this case.
When docked, a tool window can be resized either vertically or horizontally (but not both) to decrease or increase its size. A tool window docked to the top or bottom border of its owner can be resized horizontally. A tool window docked to the left or right border can be resized vertically.
When the user double-clicks a floating tool window's titlebar, or a docked window's gripper bar, the tool window is toggled from floating to docked, or vice versa.
Our docking library keeps track of whether a tool window is docked or floating. And if it is docked, we need to know to which of the owner's borders the tool window is docked. The uDockedState field of the DOCKINFO is used to store this state. As mentioned, if this field is OR'ed with DWS_FLOATING, then the tool window is floating. If not OR'ed with DWS_FLOATING, then the tool window is docked, and the remaining bits of the field are either DWS_DOCKED_LEFT, DWS_DOCKED_RIGHT, DWS_DOCKED_TOP, or DWS_DOCKED_BOTTOM depending upon to which border the tool window is docked.

We need to be able to toggle a tool window between being a child window (docked) and being a popup window (floating). This is basically accomplished with the code shown below.

// Assume "dwp" is a pointer to the tool window's DOCKINFO.

DWORD dwStyle = GetWindowLong(dwp->hwnd, GWL_STYLE);

// Is the window currently floating?
if (dwp->uDockedState & DWS_FLOATING)
{
// Toggle from WS_POPUP to WS_CHILD. We do this by altering
// the window's style flags to remove WS_POPUP, and add
// WS_CHILD. Then, we set the owner window as the parent.
SetWindowLong(dwp->hwnd, GWL_STYLE, (dwStyle & ~WS_POPUP) | WS_CHILD);
SetParent(dwp->hwnd, dwp->container);
}
else
{
// Toggle from WS_CHILD to WS_POPUP. We do this by altering
// the window's style flags to remove WS_CHILD, and add
// WS_POPUP. Then, we make sure it has no parent.
SetWindowLong(dwp->hwnd, GWL_STYLE, (dwStyle & ~WS_CHILD) | WS_POPUP);
SetParent(dwp->hwnd, NULL);
}

Look at the second SetParent API call in the code above. The only way to make a child (docked) window into a popup (floating) window is to set its parent window to zero (NULL). Because the tool window no longer has a parent, it is not visually confined to some other window. It can float freely around the desktop. But because it still has an owner window, the operating system keeps it floating above that owner window. In other words, when a window is docked, its owner window is also its parent window. When a window is floating, its owner window is no longer its parent as well.

Floating versus docked size
As mentioned, a tool window can be in one of two states: docked, or floating (undocked). We will remember the size of a tool window both in its floating state, and its docked state, and store this information in the DOCKINFO. In this way, the end user can give the tool window different sizes for its two states. Because we also allow the end user to quickly toggle between the two states by double-clicking on the gripper/titlebar, we need to remember where the tool window was last positioned in both states.

When a tool window is floating, it can be resized just like a normal window. This means that we will need to store both the width and height in the DOCKINFO. And of course, in order to remember its position, we need to store its X and Y position (in screen coordinates). These values are stored in the DOCKINFO's cxFloating, cyFloating, xpos, and ypos fields respectively.

Note: cxFloating and cyFloating are actually set to the size of the floating tool window's client (inner) area instead of the physical size of the tool window itself (including its titlebar and borders). This is because we always want the client area to remain the same size, even when the system settings change (i.e. the titlebar height is modified using the Control Panel).

When a tool window is docked, it can be resized in only one direction -- vertically or horizontally. This means that we need to remember only its width or height, but not both. If the tool window is docked to the top or bottom border of the owner, then we remember its height. If the tool window is docked to the left or right border of the owner, then we remember its width. Whichever value we remember, we store it in DOCKINFO's nDockedSize field. As far as its position is concerned, that is already remembered in the DOCKINFO's uDockedState field.

Moving a window with a drag-rectangle
The first obstacle we encounter is getting Windows to show a feedback rectangle when the end user moves a floating window around. Starting with Windows 95, a new user-interface feature was introduced. This feature is normally referred to as "Show window contents while dragging". When enabled, windows are no longer moved and sized using the standard feedback rectangle.

Unfortunately, there is no way to turn this feature off for specific windows. The SystemParametersInfo API call (with the SPI_GETDRAGFULLWINDOWS setting) can turn this feature on and off, but this is a system-wide setting, and is not really suitable. Of course, we could devise a method where we temporarily turn off the drag-window system setting just during the window movement (actually, this would be very straight-forward). The point is, it's a bit of a hack, and I prefer proper solutions to problems like this.

The only solution is to override the standard Windows behaviour and manually provide a feedback rectangle. This means processing a few mouse messages. Now, I don't want to show any code - again, the source code clearly demonstrates how to get this working (in the window procedure for a tool window, dockWndProc). What I will do is give a basic outline of the processing that is required.

The most important task is to stop the user from dragging the window around with the mouse. I know this sounds counter-productive, but we need to completely take over the standard window movement logic. This is actually quite simple - our docking window procedure just needs to handle WM_NCLBUTTONDOWN, and return 0 if the mouse is clicked in the caption area. By preventing the default window procedure from handling this message, window dragging is completely disabled.

In order to simulate the window being moved, we need to handle a few mouse messages. Only three need processing:

WM_NCLBUTTONDOWN - This message is received when the end user clicks on a tool window. In addition to returning 0 to prevent the operating system from doing normal window dragging, we draw the drag-rectangle at its initial position, and set the mouse capture using the SetCapture API call. We also install a keyboard hook so we can check if the end user presses the CTRL key (to force the tool window floating) or the ESC key (to abort the operation).
WM_MOUSEMOVE - This message is received whenever the mouse is moved. Our response is to redraw the drag-rectangle in the new position (erase it in the old position and draw it in the new position). In addition, we need to decide what type of rectangle to draw, depending on if the end user has moved the rectangle into a dockable region, or not.
WM_LBUTTONUP - This message is received when the mouse is released. We remove the drag-rectangle from the screen, release the mouse capture, and then take the appropriate action to physically reposition the tool window. This may mean docking / undocking, or simply moving the window if it was already floating.
As you can see, there's a little bit of work involved, but nothing particularly complicated. The big advantage of using this method is that the same mouse code can be used when the window is docked or floating. This keeps the code short and simple.

Drawing a drag-rectangle
A drag-rectangle is basically just a simple rectangle. This rectangle ideally needs to be drawn using XOR blitting logic, so that we can easily draw / erase the rectangle as it is moving around.

A tool window being dragged. You can see the drag rectangle.

The code below draws a shaded rectangle with the specified coordinates. The equivalent function in the source code does a little more than the code below (it draws both types of drag-rectangles), but I've stripped it down to keep it simple.

Collapsevoid DrawXorFrame(int x, int y, int width, int height)
{
// Raw bits for bitmap - enough for an 8x8 monochrome image
static WORD _dotPatternBmp1[] =
{
0x00aa, 0x0055, 0x00aa, 0x0055, 0x00aa, 0x0055, 0x00aa, 0x0055
};

HBITMAP hbm;
HBRUSH hbr;
HANDLE hbrushOld;
WORD *bitmap;

int border = 3;

HDC hdc = GetDC(0);

// Create a patterned bitmap to draw the borders
hbm = CreateBitmap(8, 8, 1, 1, _dotPatternBmp1);
hbr = CreatePatternBrush(hbm);

hbrushOld = SelectObject(hdc, hbr);

// Draw the rectangle in four stages - top, right, bottom, left
PatBlt(hdc, x+border, y, width-border, border, PATINVERT);
PatBlt(hdc, x+width-border, y+border, border, height-border, PATINVERT);
PatBlt(hdc, x, y+height-border, width-border, border, PATINVERT);
PatBlt(hdc, x, y, border, height-border, PATINVERT);

// Clean up
SelectObject(hdc, hbrushOld);
DeleteObject(hbr);
DeleteObject(hbm);

ReleaseDC(0, hdc);
}

As you can see, we have the bitmap data for our rectangle as global data in our docking library. And we simply call some graphics functions to blit in onto the screen (in a rectangular shape) at the screen position where the end user has currently moved the mouse.

Redrawing the docked windows
When a tool window's state changes from docked to floating, or vice versa, this means that the layout of the owner window needs to be redrawn. For example, if a tool window was floating, and then is docked to the owner window, then other tool windows already docked may need to be resized/repositioned to accommodate the new docked tool window.

And if a tool window was docked to the owner window, but is torn off and left floating, that means the other, remaining docked windows may likewise need to be resized/repositioned to fill the "hole" left by the formerly docked window.

Whenever a tool window's state toggles between states, our docking library has a function named updateLayout that is called to send a spoofed WM_SIZE message to the owner window to inform it that it needs to redraw itself. The owner window then is expected to redraw its contents and call a docking library function named DockingArrangeWindows. DockingArrangeWindows does all the work of repositioning and redrawing the docked tool windows.

Enumerating tool windows
In the above code excerpts, we had a placeholder function named DockingNextToolWindow that enumerated the tool windows for a given owner window. We don't actually have such a function in the docking library. Let's examine how our docking library actually enumerates tool windows.

Unfortunately, the Windows operating system does not have a function to enumerate all the windows owned by a particular window. If it did, we could just pass our owner window to that function. What the operating system does have is a function called EnumChildWindows. This enumerates all of the child windows of a given parent window. Since a docked tool window has its owner window as its parent also, EnumChildWindows will enumerate all the docked tool windows for a given owner. But EnumChildWindows will not enumerate any of the floating tool windows, because the owner window is not also the parent of the floating tool windows.

There is another operating system function called EnumWindows. This enumerates all of the top-level (i.e., popup) windows on the desktop. Since our floating tool windows are WS_POPUP style, this works to enumerate them. (But, it will enumerate all windows on the desktop in addition to our tool windows, so we have a little extra work to do to isolate only the desired tool windows). EnumWindows does not enumerate any of the children (WS_CHILD windows) of those top-level windows. So, EnumWindows will not enumerate any docked tool windows.

Therefore, enumerating all the tool windows will be a two-step process. First, we'll call EnumChildWindows to enumerate the docked windows for a given parent window (which also happens to be the owner window). Then, we will call EnumWindows to enumerate the floating tool windows for a given owner window, and do some extra processing to make sure that the windows we isolate are for the desired owner window.

Let's examine a function that counts how many total tool windows a given owner window has, both floating and docked.

Collapsetypedef struct {
UINT count;
HWND container;
} DOCKCOUNTPARAMS;

/***************** DockingCountFrames() *****************
* Counts the number of tool windows for the specified
* owner window.
*
* container = Handle to owner window.
*/

UINT WINAPI DockingCountFrames(HWND container)
{
DOCKCOUNTPARAMS dockCount;

// Initialize count to 0, and store the desired owner window
dockCount.count = 0;
dockCount.container = container;

// Enumerate/count the floating tool windows
EnumWindows(countProc, (LPARAM)&dockCount);

// Enumerate/count the docked tool windows
EnumChildWindows(container, countProc, (LPARAM)&dockCount);

// Return the total count
return dockCount.count;
}

/******************* countProc() ********************
* This is called by EnumChildWindows or EnumWindows
* for each window.
*
* hwnd = Handle of a window.
* lParam = The LPARAM arg we passed to EnumChildWindows
* or EnumWindows. That would be our DOCKCOUNTPARAMS.
*/

static BOOL CALLBACK countProc(HWND hwnd, LPARAM lParam)
{
DOCKINFO * dwp;

// Is this one of the tool windows for the desired owner window?
if (GetClassWord(hwnd, GCW_ATOM) == DockingFrameAtom &&
(dwp = (DOCKINFO *)GetWindowLong(hwnd, GWL_USERDATA)) &&
dwp->container == ((DOCKCOUNTPARAMS *)lParam)->container)
{
// Yes it is. Increment count.
((DOCKCOUNTPARAMS *)lParam)->count += 1;
}

// Tell operating system to continue.
return TRUE;
}

Notice that we use countProc() as the callback for both EnumWindows and EnumChildWindows. And we pass our own initialized DOCKCOUNTPARAMS structure to our callback. First, we call EnumWindows to enumerate the floating windows. Then we call EnumChildWindows to enumerate the docked windows for our desired owner. So, let's examine countProc(). The entire key to making this work is to fetch and check the class ATOM for the window. If it matches the ATOM we got when we registered our own docking window class (returned by RegisterClassEx), then we know this is one of our tool windows. And if it is one of our tool windows, we know that its GWL_USERDATA field should contain its DOCKINFO. And note that the owner window handle has been stored in the DOCKINFO's container field. So we need only compare this handle with the owner handle passed to DockingCountFrames in order to determine if it is a tool window for the desired owner window.

Various other features of the library
The discussion above details all of the most important aspects of our docking library's features. But, there are some more, incidental features which are optional. You can enable any of these features for a given tool window just by setting the appropriate value into its DOCKINFO's dwStyle field. For example, you can force a tool window to always stay docked or floating. You can force it to keep its original size. You can restrict to which sides of the owner window the tool window may be docked.

When DockingAlloc creates a DOCKINFO, none of these extra features are enabled.

--------------------------------------------------------------------------------

An application
Up to this point, we've discussed only the code in the docking library. Since the whole intent of the library is to be used by an application, now we'll turn our attention to a sample application. There is a sample C application called DockTest included with the library. This example creates one owner window. The owner has a View -> Tool Window menu item you can select to create a tool window. You can then move the tool window around, docking and undocking it, to get a feel for how the implementation works. Each time you select this menu item, another tool window is created, so you can see how multiple tool windows can be floated and docked.

The owner window we create is an MDI window, and its window procedure is frameWndProc. You can open a document window with the File -> New menu item, and see how the docked tool windows interact with a document window. (But, as we'll see later, the application needs to do a little work to manage this interaction.)

Creating a tool window
Let's examine how the application creates a tool window. This happens when the View -> Tool Window menu item is selected, so the place where we create the tool window is in frameWndProc's handling of WM_COMMAND for menu ID IDM_VIEW_TOOLWINDOW. Below is a slightly simplified version of what needs to be done to create a tool window:

Collapsevoid createToolWindow(HWND owner)
{
DOCKINFO *dw;
HWND frame;

// Allocate a DOCKINFO structure.
if ((dw = DockingAlloc(DWS_DOCKED_BOTTOM)))
{
// Create a Docking Frame window (ie, the tool window).
if ((frame = DockingCreateFrame(dw, owner, "My title")))
{
// Create the child window that will be hosted inside of the Docking
// Frame window (ie, the contents of the tool window's client area)
// and save it in the DOCKINFO's focusWindow field. We'll create an
// EDIT control to be the contents, but you can utilize any standard
// control, or a child window of your own class.
if((dw->focusWindow = CreateWindow("EDIT", 0,
ES_MULTILINE|WS_VSCROLL|WS_CHILD|WS_VISIBLE,
0,0,0,0,
frame,
(HMENU)IDC_MYEDIT, GetModuleHandle(0), 0)))
{
// Show the Docking Frame.
DockingShowFrame(dw);

// Success!
return;
}

// Destroy the tool window if we can't create its contents.
// NOTE: The docking library will free the above DOCKINFO.
DestroyWindow(frame);
}
MessageBox(0, "Can't create tool window", "ERROR", MB_OK);
}
else
MessageBox(0, "No memory for a DOCKINFO", "ERROR", MB_OK);
}

First, we call DockingAlloc to get a DOCKINFO structure. We pass the desired initial state, which will be one of DWS_FLOATING, DWS_DOCKED_LEFT, DWS_DOCKED_RIGHT, DWS_DOCKED_TOP, or DWS_DOCKED_BOTTOM, depending upon whether we want the tool window initially created floating, or docked to one of the four borders. The docking library creates a DOCKINFO and initializes it to default values, returning a pointer.

At this point, we could modify the DOCKINFO if we want something other than the default features. In the above code, we simply go with the defaults.

Next, we call DockingCreateFrame to create the actual tool window. We pass the DOCKINFO we just got, the handle to our owner window, and the desired title for the tool window (which is shown only when the tool window is floating). DockingCreateFrame will create the tool window and return its handle. The tool window is not created visible, so nothing has yet shown up onscreen.

Now, a tool window with nothing inside of it would not be of much use. So we need to create something inside of the tool window that is of use to the end user. We can say that the tool window needs some "contents". Specifically, we need to create some WS_CHILD window which has the tool window as its parent. This can be any standard control, such as an Edit box, list box, tree-view control, etc. Or it could be a window of our own class. In the above code, we simply create a multi-line Edit control. Note that we have set the tool window to be this control's parent, and also specified the WS_CHILD style. This will cause the Edit control to be visually embedded inside of the tool window, and automatically move with the tool window. The size and position of the Edit control is not important now, because it will be sized and positioned later, before the tool window is finally made visible. We stuff the handle to this control into the DOCKINFO's focusWindow field. The docking library will automatically size this control to fill the client area of the tool window, and also give the control the focus whenever the user activates that tool window.

Finally, we call DockingShowFrame. This first sends a WM_SIZE message to our owner window (which is where we will do the final sizing/positioning of the tool window and its contents-window), and then makes the tool window visible.

That's all there is to creating a tool window. At this point, the docking library will manage the docking and undocking of this window.

Handling WM_SIZE message in the owner
The docking library transparently handles most aspects of the tool windows. But there are a couple times when it needs help from the application. One such time is whenever the owner window is resized. Given a new size for the owner window, it stands to reason that any docked tool window may also need to be resized and repositioned so that it stays docked to the desired side of the owner window. For this reason, the owner window will have to do the following when it receives a WM_SIZE:

Collapse case WM_SIZE:
{
HDWP hdwp;
RECT rect;

// Do the default handling of this message.
DefFrameProc(hwnd, MainWindow, msg, wParam, lParam);

// Set the area where tool windows are allowed.
// (Take into account any status bar, toolbar etc).
rect.left = rect.top = 0;
rect.right = LOWORD(lParam);
rect.bottom = HIWORD(lParam);

// Allocate enough space for all tool windows which are docked.
hdwp = BeginDeferWindowPos(DockingCountFrames(hwnd,
1) + 1); // + 1 for the MDI client

// Position the docked tool windows for this owner
// window. rect will be modified to contain the "inner" client
// rectangle, where we can position an MDI client.
DockingArrangeWindows(hwnd, hdwp, &rect);

// Here we resize our MDI client window so that it fits into the area
// described by "rect". Do not let it extend outside of this
// area or it (and the client windows inside of it) will be obscured
// by docked tool windows (or vice versa).
DeferWindowPos(hdwp, MainWindow, 0, rect.left, rect.top,
rect.right - rect.left, rect.bottom - rect.top,
SWP_NOACTIVATE|SWP_NOZORDER);

EndDeferWindowPos(hdwp);

return 0;
}

First, we pass the WM_SIZE to DefFrameProc to let the operating system do the default sizing of the owner window. We need to do this first so that the owner window's size is finalized before we go ahead and resize/reposition the docked tool windows.

The docking library has a function called DockingArrangeWindows that redraws all of the docked windows for a given owner. So to completely redraw its docked windows, all the owner needs to do is call this one function. But, there are a couple prerequisites. First, the owner must fill in a RECT with the dimensions of its client area, and pass this to DockingArrangeWindows. Secondly, the owner window must call the Windows API BeginDeferWindowPos to reserve enough space for all the docked windows. (The docking library has a function called DockingCountFrames which can be called to retrieve the total number of docked tool windows in an owner.) We use BeginDeferWindowPos so that, if there are many tool windows, we defer the final painting until after all of them are sized and positioned. This is more efficient and doesn't cause any unsightly visual artifacts for the end user to witness.

One very important aspect to note is that, after DockingArrangeWindows resizes and repositions all of the docked tool windows, it updates the RECT so that it encompasses only the owner client area not occupied by the tool windows. In other words, the RECT is the remaining, blank client area. We take this remaining area, and we resize/reposition our MDI child so that it fills only this remaining area. In this way, our document windows are not obscured by docked tool windows, and vice versa.

Handling WM_NCACTIVATE message in the owner
Another instance where our docking library needs help from the application is whenever the owner window receives a WM_NCACTIVATE message. Remember earlier we discussed how to keep all tool windows' titlebar activation in sync with the owner window. Now, we need the owner window to let the docking library know whenever it receives a WM_NCACTIVATE message. The owner window will need to do the following:

case WM_NCACTIVATE:
{
DOCKPARAMS dockParams;

dockParams.container = dockParams.hwnd = hwnd;
dockParams.wParam = wParam;
dockParams.lParam = lParam;
return(DockingActivate(&dockParams));
}

We simply fill in a DOCKPARAMS struct (defined in DockWnd.h) with the values we receive from the WM_ACTIVATE message, and also fill in the owner window handle (in DOCKPARAMS container field) and the handle of the window receiving the WM_ACTIVATE (which here, is the owner window). Then we call DockingActivate which takes care of syncing all tool windows' titlebar activation.

Handling WM_ENABLE message in the owner
Another instance where our docking library needs help from the application is whenever the owner window receives a WM_ENABLE message. Remember earlier we discussed how to keep all tool windows' enabled state in sync with the owner window. Now, we need the owner window to let the docking library know whenever it receives a WM_ENABLE message. The owner window will need to do the following:

case WM_ENABLE:
{
DOCKPARAMS dockParams;

dockParams.container = dockParams.hwnd = hwnd;
dockParams.wParam = wParam;
dockParams.lParam = lParam;
return(DockingEnable(&dockParams));
}

This is almost the same as the WM_NCACTIVATE handling, but it concerns the WM_ENABLE message, and we call a function named DockingEnable. DockingEnable takes care of syncing all tool windows' enabled state.

Handling messages sent to standard controls
You'll note that we used a standard Edit control as the content of our tool window. As you should know, an Edit window sends messages to its parent for certain actions. For example, when the end user alters the contents of the Edit control, a WM_COMMAND message is sent with a notification code of EN_CHANGE.

But remember that the parent window is the tool window, and our tool window procedure (dockWndProc) is in the docking library. So how does an application get a hold of that message?

There is a DockMsg field in the DOCKINFO. Into this field, we will stuff a pointer to a function in our application. We do this after we DockAlloc the DOCKINFO as so:

// Allocate a DOCKINFO structure.
if ((dw = DockingAlloc(DWS_DOCKED_BOTTOM)))
{
// Set our own function for the docking library to call.
dw->DockMsg = myMessages;

...

Whenever dockWndProc receives a message that it doesn't handle, such as a WM_COMMAND or WM_NOTIFY, it will call our application function, passing the DOCKINFO of the tool window, as well as the message, WPARAM, and LPARAM parameters.

Here is an example of the function we could add to handle a EN_CHANGE from our IDC_MYEDIT edit control:

LRESULT WINAPI myMessages(DOCKINFO * dwp, UINT message, WPARAM wParam, LPARAM lParam)
{
switch (message)
{
case WM_COMMAND:
{
if (LOWORD(wParam) == IDC_MYEDIT)
{
if (HIWORD(wParam) == EN_CHANGE)
{
// Here we would handle the EN_CHANGE, and then return
// 0 to tell the docking library we handled it.
return 0;
}
}
}
}

// Return -1 if we want the docking library to do default handling.
return -1;
}

Note: Each DOCKINFO can have its own DockMsg function, so you do not need to worry about control ID conflicts between tool windows.

Multiple child windows inside a tool window
Above, we used a single Edit control to fill the client area of the tool window. But what if we would like several controls inside the tool window, for example, an Edit control as well as a push button labeled "Clear" which clears the text from the Edit control?

This is entirely possible, but there are a couple of prerequisites. First, when we create the Edit and button controls, we must make both of them children of the tool window. Secondly, we must provide a function that will resize and reposition the controls, and stuff a pointer to this in the DOCKINFO's DockResize field after we DockAlloc the DOCKINFO.

// Allocate a DOCKINFO structure.
if ((dw = DockingAlloc(DWS_DOCKED_BOTTOM)))
{
// Set our own functions for the docking library to call.
dw->DockMsg = myMessages;
dw->DockResize = myResize;

...

When the docking library calls our function, it passes the DOCKINFO for the tool window, as well as a RECT that encompasses the area that we need to fill. Here is an example of a function we could add to resize and reposition the edit and button controls so that the button stays near the bottom border of the tool window, and the edit control fills the rest of the area:

void WINAPI< myResize(DOCKINFO * dwp, RECT * area)
{
HWND child;

// Position the button above the bottom border
child = GetDlgItem(dw->hwnd, IDC_MYBUTTON);
SetWindowPos(child, 0, rect->left + 10,
rect->bottom - 20, 50, 18, SWP_NOZORDER|SWP_NOACTIVATE);

// Let the edit fill the remaining area
child = GetDlgItem(dw->hwnd, IDC_MYEDIT);
SetWindowPos(child, 0, rect->left, rect->top,
rect->right - rect->left, (rect->bottom - rect->top) - 22,

SWP_NOZORDER|SWP_NOACTIVATE);
}

Closing a tool window
When an owner window is destroyed, all of its tool windows are also automatically destroyed (except if you use the DWS_FREEFLOAT style. In that case, your owner window should handle WM_DESTROY and call DockingDestroyFreeFloat). The docking library will normally free the DOCKINFO for each tool window destroyed.

If you wish to manually close a tool window, you simply call the Windows API DestroyWindow, passing the handle to the desired tool window. Again, the docking library will normally free the DOCKINFO.

If you wish to override the docking library's default behavior of freeing the DOCKINFO, then you must write your own function, and stuff a pointer into the DOCKINFO's DockDestroy field. The docking library will call this function (passing it the DOCKINFO) whenever the tool window has been destroyed. It is your responsibility to eventually free the DOCKINFO by passing it to DockingFree. One use for this is to keep a DOCKINFO allocated (for a given tool window) throughout the lifetime of your application. You will reuse this same DOCKINFO with DockingCreateFrame each time the end user reopens that tool window. Because the DOCKINFO stores the last size and position of the tool window, this means that the tool window will reappear where it was located right before it was previously destroyed. You will then free the DOCKINFO only when your application is ready to terminate.

Saving/Restoring a tool window size/position
Each time that you run your application, you will normally want to restore the same sizes and positions for the tool windows that the end user set upon the last time your application was run. The docking library has two functions to help save and restore tool window positions. The data is saved to the Windows registry under some key of your choosing. The size/position of each tool window is saved separately as values under that key.

Before you free a tool window's DOCKINFO, you should first create/open some registry key of your choosing in order to save that tool window's settings. Then, you will pass the DOCKINFO, and a handle to this open key, to DockingSavePlacement. The docking library will save that tool window's settings. Here is an example of how we could save the settings of a tool window under the registry key "HKEY_CURRENT_USER\Software\MyKey\MyToolWindow":

HKEY hKey;
DWORD temp;

// Open/Create the "Software\MyKey\MyToolWindow" key under CURRENT_USER.
if (!RegCreateKeyEx(HKEY_CURRENT_USER, "Software\\MyKey\\MyToolWindow",
0, 0, REG_OPTION_NON_VOLATILE, KEY_ALL_ACCESS, 0, &hKey, &temp))
{
// Let the docking library save this tool window's settings.
DockingSavePlacement(dw, hKey);

// Close registry key.
RegCloseKey(hKey);
}

Whenever your program runs, it should restore those settings by calling DockingLoadPlacement right after DockAlloc'ing the DOCKINFO for that tool window. Here is an example of restoring the previously saved settings:

// Allocate a DOCKINFO structure.
if ((dw = DockingAlloc(DWS_DOCKED_BOTTOM)))
{
HKEY hKey;

// Open the "Software\\MyKey\\MyToolWindow" key under CURRENT_USER.
if (!RegOpenKeyEx(HKEY_CURRENT_USER, "Software\\MyKey\\MyToolWindow",
o, KEY_ALL_ACCESS, &hKey))
{
// Let the docking library restore this tool window's settings.
DockingLoadPlacement(dw, hKey);

// Close registry key.
RegCloseKey(hKey);
}

...

Note: If there are no previously saved settings for the tool window, then none of the DOCKINFO fields are altered

CWaitingTreeCtrl

2008-01-11T08:30:00.000-08:00

What is it & Why?
My real intention, when I started this project, was to develop a tree control to display network resources. Enumerating network resources could be a lengthy operation, so I decided to actually do enumeration only when the user tries to expand an item. I realized soon that this kind of behavior could be useful to many other purposes, so I put all the code needed to have this behavior in a base class for my tree control, that was ready to use with other classes. I also added support for visual feedback, while an item is being populated.

When the user clicks to expand an item, a new item is added which contains a wait message. If you implement some animation inside the tree control the message is visible, otherwise it remains hidden. The blue rectangle is the area in which you can draw: you can do everything you want, even completely overwrite the wait message, as long as you stay within that area. Timed animations are also supported.

In the meantime, you can populate the item being expanded with new child items and the tree control will redraw itself to reflect the new content only when you have finished: the user will see only your animation. You may also choose not to display anything while populating, this is very easy.

I also added the ability to repaint the control while populating. The main thread is busy until the population process completes, this is "by design", but I noticed that it still receives WM_ERASEBKGND messages, at least on Win2K. So now I take a snapshot of the control before the process takes place and use that bitmap to redraw the control's background until the expanded item gets populated completely. (Please, report if this works on other platforms)

If you want to see some examples of what you can do with this class take a look at these articles:

CNetworkTreeCtrl
CShellTreeCtrl
CProgressFX and CHourglassFX
Using the Class
This class provides a common interface for two types of derived classes: those providing animation effects and those providing the items for the tree control. Obviously you can implement a single class that provides both content and animation, but you may lose in code re-use.

If you keep the two implementations separated, you can:

focus your attention on the content provider class, which will handle the underlying tree control
choose a ready-to-use animation provider class to add visual feedback to your content provider class or create your own
leave the control without any animation (users won't see the wait message)
enable the wait message when you use your content provider class
Providing Tree Content
Just derive your class from CTreeCtrl and then replace each occurence of CTreeCtrl with CWaitingTreeCtrl in both your '.h' and '.cpp' files.

class CMyTreeCtrl : public CWaitingTreeCtrl
{
...
};

Tip: if you want to use Class Wizard to generate message handlers later, you may temporarily change the base class in the BEGIN_MESSAGE_MAP macro to CTreeCtrl, but remember to restore it to the correct one (CWaitingTreeCtrl) and to change every call to the base implementation in the generated message handlers.

BEGIN_MESSAGE_MAP(CMyTreeCtrl, CWaitingTreeCtrl)
//{{AFX_MSG_MAP(CMyTreeCtrl)
...
//}}AFX_MSG_MAP
END_MESSAGE_MAP()

To fill in the tree control with items you are required to override PopulateItem and to call PopulateRoot somewhere to insert items at the first level. You may also override WantsRefresh to automatically refresh some items whenever they are expanded by user.

CWaitingTreeCtrl::PopulateItem
virtual void PopulateItem(HTREEITEM hParent)

This function gets called in the main thread after an item is expanded and the wait message is displayed. The animation takes place during the execution of this function and it is stopped when the function returns.

Tipically, you use the hParent parameter to get information about the parent item to populate and then you start adding new items. No redraw will happen until this function has finished, this is by design.

Also, there's currently no mean to abort this function, because it blocks the main thread, but you may create the underlying tree control in another user interface thread and provide your own means.

When you implement this function you have to remember the following rules:

you are supposed to add only one level of child items and you should only work with the hParent item and its children
you can always suppose that the hParent item has no children when the function gets called
you need to handle the special case in which hParent is TVI_ROOT
you should set the cChildren field of the TVITEM structure to 1 if you want that child item to be expandable
you should notify your progress to the base class (see below)
Returning TRUE means that the base class will check if you added any items and if it doesn't find any you will not be able to expand the hParent item again (the plus button will be removed).

Returning FALSE means that the hParent item will be always expandable, even if it has no children now. Note that it doesn't mean that the hParent item will be refreshed every time it's expanded, but that it will be expandable until it gets some children. If you want to refresh the hParent item each time it's expanded you have to override WantsRefresh.

Note: if you associate some data with each item you will probably handle the TVN_DELETEITEM notification. If you do so, you should ignore items that have the lParam field equals to zero. They could be the items used to display the wait message and they have no associated data. However this should not be a problem, since a zero value is tipically a NULL pointer.

CWaitingTreeCtrl::WantsRefresh
virtual BOOL WantsRefresh(HTREEITEM hParent)

This function gets called just before PopulateItem, only if the hParent item already has children, to ask a derived class if it wants the item's children to be refreshed.

You have to override this function if you want an item to refresh its children whenever the user expand it. Remember that to refresh items that have no children, you have to tell the base class not to check for inserted items by returning FALSE in your PopulateItem override.

Return TRUE if you want to automatically refresh the item (in the sense described above), or FALSE if you don't want automatic refresh. The base class implementation simply returns FALSE.

CWaitingTreeCtrl::PopulateRoot
void PopulateRoot()

You have to call this function somewhere in your derived class, if you want to see some items. A good place could be in your PreSubclassWindow or OnCreate override, but you may decide to have a function which initializes some data associated with the root item and then simulates a root expansion, populating the first level of items (PopulateItem is called with TVI_ROOT as the parent item).

You may also populate the first or some deeper levels of the tree without requiring the user to expand any items. A parent item that already has children won't be passed to PopulateItem, unless WantsRefresh says it must be refreshed. So static items are perfectly legal and can be used in conjunction with dynamic items.

CWaitingTreeCtrl::SetPopulationCount
void SetPopulationCount(int iMaxSubItems, int iFirstSubItem = 0)

You should call this function in your PopulateItem override, before adding any item, to set the total count of items you plan to insert (iMaxSubItems) and the optional initial value (iFirstSubItem).

You should set the total count to zero if you don't know how many items you're going to insert.

CWaitingTreeCtrl::IncreasePopulation
void IncreasePopulation(int iSubItemsToAdd = 1)

You should call this function in your PopulateItem override, when you insert a new item or a group of items. The current count is increased of the iSubItemsToAdd parameter, which can be a negative value.

CWaitingTreeCtrl::IncreasePopulation
void UpdatePopulation(int iSubItems)

You should call this function in your PopulateItem override, when you insert a new item or a group of items. The current count is updated to the value of iSubItems.

Providing Animation
Create a generic class and derive it from CWaitingTreeCtrl. Later you may replace the base class with a template, so that you can use the class as a plug-in to add custom animations and visual effects to a generic CWaitingTreeCtrl-derived class. (I'm not familiar with templates, so there could be a better way to do it).

template
class CMyAnimationFX : public BASE_TYPE
{
...
};

There are two types of animations: those refreshing at specified time intervals and those getting updated only when new items are inserted by a content provider class.

To display an animation frame, you have to override DoAnimation. If you want to initialize some variables related to the message item, such as the position of the animation frames, you need to override PreAnimation and PostAnimation (only if you need to free some memory at the end).

In all these functions you should call the base class implementation, especially when not directly deriving from CWaitingTreeCtrl. This way, and using templates, you can add more than a single animation to your content provider class.

CWaitingTreeCtrl::PreAnimation
virtual void PreAnimation(HTREEITEM hItemMsg)

This function gets called in the main thread just before the animation starts. Its only argument is the handle of the tree item displaying the wait message. You may use it to calculate the position in which to show the animation, but you shouldn't store it for later use. You can't assume the item still exists during the animation.

The wait message item is not visible by default, if you want to use its attributes to display something near to it, such as an animated item's image, or if you change its visual aspect, you have to call ShowWaitMessage in the constructor (or somewhere before entering this function).

You usually implement this function to initialize additional variables you may need during the animation. Remember that the hItemMsg item is a temporary item: you can use it only from inside this function, and only if the wait message is visible.

If you use the hItemMsg to draw something, you need to make it visible. If the message item is visible, you may also change the visual aspect of the tree control, such as scrolling the client area to properly displaying the animation. If the wait message is hidden the tree control can't redraw itself. You may have this situation if you don't draw the animation inside the tree control, but instead you use another window or control.

CWaitingTreeCtrl::DoAnimation
virtual void DoAnimation(BOOL bTimerEvent, int iMaxSteps, int iStep)

This function gets called in a higher priority worker thread each time you need to update your animation. If the bTimerEvent argument is TRUE this function has been called when the timer interval elapsed, otherwise the item count has been updated. If you want to handle timer events you have to specify the timer period calling SetAnimationDelay before the animation starts, either in the constructor or in PreAnimation.

The other two arguments reflect the current progress of the enumeration process: iMaxSteps is the total number of steps of the process (zero means unknown), while iStep is the current step. You may choose to ignore these during a timer event. You tipically use this information to display a progress bar.

You need to implement this function to draw a new frame of your animation. If you draw within the tree control, you should only draw inside the rectangle obtained by calling GetItemRect(hItemMsg, lpRect, FALSE). Remember that you can't call this function here: you may use CTreeCtrl::GetItemRect or CWaitingTreeCtrl::GetItemImageRect only inside PreAnimation.

CWaitingTreeCtrl::PostAnimation
virtual void PostAnimation()

This function gets called in the main thread just after the animation ends.

You have to implement this function only if you need to clean up some additional variables.

CWaitingTreeCtrl::SetAnimationDelay
void SetAnimationDelay(UINT nMilliseconds)

You may call this function in the constructor, or in your PreAnimation override, to set the delay between timer events. A delay of zero means you don't need timer events and it is the initial value. If you want timer events you have to explicitly call this function.

Note that there is only one timer, so the last call to this function before the animation starts takes precedence (if you want to use multiple animation classes).

CWaitingTreeCtrl::GetItemImageRect
BOOL GetItemImageRect(HTREEITEM hItem, LPRECT pRect)

You may call this function if you need to draw over an item's image, usually beside the wait message.

The return value is TRUE if the function is successful, FALSE otherwise.

Public Functions
There are also a few public functions that your content provider class will inherit.

CWaitingTreeCtrl::SetWaitMessage
void SetWaitMessage(LPCTSTR pszText, HICON hIcon = NULL)

You may call this function to change the wait message's text and image. If the hIcon argument is NULL the item gets a blank icon.

CWaitingTreeCtrl::ShowWaitMessage
void ShowWaitMessage()

You need to call this function in your animation provider class, either in the constructor or in any place before PreAnimation gets called, if you want to draw within the tree control. You are supposed to draw only in the rectangular area occupied by the wait message, so it must be visible. If you draw your animation in another window or if you use another control to provide visual feedback, you are not required to make the wait message visible.

You may also call this function if you don't have any animations, but you want the static wait message. Never call this function in your content provider class, the choice to display the message is left to the final user of your tree control.

Remember that the animation provider class should always assume that the wait message is not visible, and so it must show it if it needs to draw inside the control.

CWaitingTreeCtrl::RefreshSubItems
void RefreshSubItems(HTREEITEM hParent)

You may call this function to manually refresh the hParent item's children. You can only refresh items that can be expanded, whether or not they actually have children. In fact you assign a button only to items that can be expanded by the user, and only those items can be refreshed.

Updates
17 Oct 2000
Initial public release.
27 Sep 2001
Fixed a bug with refreshing empty items and possibly some other things
Added background repainting while busy (at least on Win2k)
License changed to Artistic License
Conclusion
This class probably needs some adjustments, but it's enough for me now. I think you may better understand this class if you see some implementations, so take a look at the other articles (see top). I will appreciate any comment, suggestion or contribution. Any help to improve this (and the others) article is welcome, I know it's not easy for you to read and it has been difficult for me to write.

License
This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

A list of licenses authors may to use can be found here

Quick Print

2008-01-11T08:29:00.000-08:00

This printing class is based on the article by Rob A. Fraydl, "Creating a Printing Class". In fact, it's just an extension of the original class. What exactly has been extended? Well, first of all, I renamed the class to QPrint (stands for quick print). Big achievement, eh? OK, OK, this feature is not an extension at all, but anyway it's a start.

The second extension was to add the ability to manage big strings that extend beyond the right border of the physical paper size. Now, the big strings are not clipped at the end of the right border, but instead they continue to the next line.

Last but not the least, the new printing class supports tables as well as page orientation.

Background
As already said in the beginning, this piece of code did not start from scratch. You can always refer to the original printing class named CPrivatePrint. Please feel free to use and improve this code. Other than the additions I have made, I also corrected some bugs. I am not sure in some points if they were bugs, especially the part with the logical units, using the macros CALCF(x), CALCY(y), and CALCX(x). I deleted CALCX(x) completely, and the CALCY(x) partially, from the source code. So, if you meet "strange" results, please let me know. The others were safe point bugs that got corrected successfully.

I tested this class using the JawsPDF Creator (to save paper, time, patience, and finally money!) and different ink jet printers. It worked flawlessly.

Using the code
I chose this printing class as a base to build QuickPrint, because it was compact and easy to use. So, the only thing that you have to do is to declare the CQPrint prt and start using it. The following example shows the basic steps that you have to do:

void TestDlg::OnPrintButton ()
{
CQPrint prt;
HPRIVATEFONT hFont;

// Step 1 : call the CPrintDialog
if (prt.Dialog() == -1)
return;

//Step 2 : Start the Print
prt.StartPrint();

//Step 3 : Create a printing font
hFont = prt.AddFontToEnvironment("Arial Greek",8,8);

//Step 4 : Start Page
prt.StartPage();

//Step 5 : The actual printing goes here
prt.Print(hFont,"Set Font Arial Size = 8",FORMAT_NORMAL);

//Step 6 : now end the page
prt.EndPage();

//Step 7 : close the print document
// and release it in the spooler
prt.EndPrint();
}

You can see a live example (like the picture in this tutorial) in the following code:

Collapse CQPrint prt;
HPRIVATEFONT hFont,hFont1,hFont2;
CSize dim;
CRect margins;

// initializing the class and printer selection
if (prt.Dialog() == -1)
return;

// create the print document in the spooler
prt.SetPageOrientation (DMORIENT_PORTRAIT);
prt.StartPrint();

// Adding the arial font to the class
hFont = prt.AddFontToEnvironment("Arial Greek",8,8);
hFont1 = prt.AddFontToEnvironment("Arial Greek",9,9);
hFont2 = prt.AddFontToEnvironment("Arial Greek",10,10);

prt.SetMargins(40,160,200,200);
prt.SetDistance(5);

prt.GetDrawDimension(dim);
prt.GetMargins(margins);
prt.SetHPos(dim.cy-margins.top);
// activate a header and footer line
prt.SetActiveFont (hFont);
prt.ActivateHF(HeaderFooter);

// set margins and line spacing

prt.StartPage();
// start printing the lines

prt.Print(hFont,"Set Font Arial Size = 8",FORMAT_NORMAL);
prt.Print(hFont,"My name is George Papaioannou",FORMAT_NORMAL);
prt.Line (PS_SOLID);

prt.Print(hFont1,"Set Font Arial Size = 9",FORMAT_NORMAL);
prt.Print(hFont1,"My name is George Papaioannou",FORMAT_NORMAL);
prt.Line (PS_SOLID);

prt.Print(hFont2,"Set Font Arial Size = 10",FORMAT_NORMAL);
prt.Print(hFont2,"My name is George Papaioannou",FORMAT_NORMAL);
prt.Line (PS_SOLID);

// ===================== TABLE 1 ==========================

prt.lf(hFont2);
prt.lf(hFont2);
prt.Print(hFont2,"TABLE 1",FORMAT_CENTER);
prt.lf(hFont2);
prt.lf(hFont2);

int size[5];
size[0]=800;
size[1]=400;
size[2]=400;
size[3]=400;

prt.SetTableColumns (3);
prt.SetTableColumnsSize(size);
prt.SetTableHorLine (TRUE);
prt.SetTableVerLine (TRUE);
prt.SetTableBorderSize (200);
prt.SetTableJustify(FORMAT_CENTER);
prt.SetTableBorderLine(5);
//set the headers
prt.SetFace (hFont,FACE_NORMALBOLD);
prt.AddTableRecord (hFont,"NAME|GRADE|CITY",FORMAT_CENTER);
//set the records
prt.SetFace (hFont,FACE_NORMAL);
prt.AddTableRecord (hFont,"George Papaioannou|Grade"
" A|Lives at Athens (Greece)",FORMAT_CENTER);
prt.AddTableRecord (hFont,"Kostas Papadimitriou|"
"Grade B|Lives at Larissa (Greece)",FORMAT_CENTER);

prt.lf(hFont2);
prt.lf(hFont2);

prt.Print(hFont2,"AppWizard has created this PrintJob"
" application for you. This application "
"not only demonstrates the basics"
" of using the Microsoft Foundation classes "
"but is also a starting point for"
" writing your application.",FORMAT_NORMAL);

// ===================== TABLE 2 ==========================
prt.lf(hFont2);
prt.lf(hFont2);
prt.Print(hFont2,"TABLE 2",FORMAT_CENTER);
prt.lf(hFont2);
prt.lf(hFont2);

size[0]=800;
size[1]=400;
size[2]=600;
size[3]=600;

prt.SetTableReset();
prt.SetTableColumns (4);
prt.SetTableColumnsSize(size);

//set the headers
prt.SetFace (hFont,FACE_NORMALBOLD);
prt.AddTableRecord (hFont,"NAME|GRADE|CITY|ADDRESS",
FORMAT_CENTER);
//set the records
prt.SetFace (hFont,FACE_NORMAL);
prt.AddTableRecord (hFont,"George Papaioannou|Grade A|"
"Athens (Greece)|23 Octobriou",FORMAT_CENTER);
prt.AddTableRecord (hFont,"Kostas Papadimitriou|Grade B|"
"Larissa (Greece)|Irron Polytechneiou",FORMAT_CENTER);

// =================== TABLE 3 ==============================
prt.lf(hFont2);
prt.lf(hFont2);
prt.Print(hFont2,"TABLE 3",FORMAT_CENTER);
prt.lf(hFont2);
prt.lf(hFont2);

size[0]=800;
size[1]=400;
size[2]=600;
size[3]=600;

prt.SetTableReset();
prt.SetTableColumns (4);
prt.SetTableColumnsSize(size);
prt.SetTableHorLine (TRUE);
prt.SetTableBorderSize (200);
prt.SetTableJustify(FORMAT_RIGHT);

//set the headers
prt.SetFace (hFont,FACE_NORMALBOLD);
prt.AddTableRecord (hFont,"NAME|GRADE|CITY|ADDRESS",FORMAT_CENTER);
//set the records
prt.SetFace (hFont,FACE_NORMAL);
prt.AddTableRecord (hFont,"George Papaioannou|Grade A|"
"Athens (Greece)|23 Octobriou",FORMAT_CENTER);
prt.AddTableRecord (hFont,"Kostas Papadimitriou|Grade B|"
"Larissa (Greece)|Irron Polytechneiou",FORMAT_CENTER);

// now end the page
prt.EndPage();

// close the print document and release it in the spooler
prt.EndPrint();

You can call the following table methods:

CQPrint::SetTableColumns(int Cols)
SetTableColumns sets the table columns.

prt.SetTableColumns(3);

CQPrint::SetTableBorderSize(int Border)
SetTableBorderSize sets the table border space. Default is 50.

prt.SetTableBorderSize(50);

CQPrint::SetTableColumnsSize(int Cols,int size[])
SetTableColumnsSize sets the table column size. The maximum number of columns permitted is 20.

int size[4];
size[0]=800;
size[1]=400;
size[2]=600;
size[3]=600;

prt.SetTableColumnsSize(4,size);

CQPrint::SetTableHorLine(BOOL HLine)
SetTableHorLine sets the horizontal lines. Default is OFF (no lines).

prt.SetTableHorLine(TRUE);

CQPrint::SetTableVerLine(BOOL HLine)
SetTableVerLine sets the vertical lines. Default is OFF (no lines).

prt.SetTableVerLine(TRUE);

CQPrint::SetTableBorderLine(int Border)
SetTableBorderLine sets the width of the border line. Default is 0 (thin line).

prt.SetTableBorderLine(10);

CQPrint::SetTableJustify(int Justification)
SetTableJustify justifies the table. Default is FORMAT_NORMAL. You can also use FORMAT_CENTER and FORMAT_RIGHT.

prt.SetTableJustify(FORMAT_CENTER);

CQPrint::SetTableReset()
SetTableReset resets all the table parameters to the default state. Use this if you don't want to change values when using consecutive tables with different visual parameters.

prt.SetTableReset();

CQPrint::AddTableRecord(HPRIVATEFONT font, CString strText, UINT Format)
AddTableRecord inserts values to the current table. Use the | symbol to change the column in strText.

prt.AddTableRecord (hFont,"Column 1|Column 2|"
"Column 3|Column 4",FORMAT_CENTER);

If you want to change the page orientation, use SetPageOrientation before StartPrint. Use the values DMORIENT_PORTRAIT or DMORIENT_LANDSCAPE.

prt.SetPageOrientation (DMORIENT_PORTRAIT);
prt.StartPrint();

You can always use the "original" methods of the class like SetMargins(), Line(), lf() etc. More details about these functions can be found in the demo. You have to experiment a little before you use it. Don't forget that this class also supports bitmap printing with the InsertBitmap() method. I was wondering if there is someone who can extend this class to support other image formats using GDI+. Maybe the next version of QPrint will support print preview as well as image printing (JPG, GIF, PCX etc.), but it's a little bit complicated, since my wife Natasa is already negative to this perspective. So, don't count on this.

Don't forget that you can always use the callback function ActivateHF() that is used to print headers and footers. I have also included this feature in my demo. Anyway, there are more methods in QPrint that you can find in this short tutorial, so start playing with the demo to get an idea of how this stuff works, and then try to discover the more advanced capabilities that this class supports.

Points of Interest
Please, if you see problems with this class, especially with the (LOGPIXELSY, LOGPIXELSX) width and height of the letters, feel free to contact with me.

History
There is no history for the Quick Print (QPrint) class. This is the first and the only version. The base and the original code for this class was from Rob A. Fraydl and is called CPrivatePrint.

License

Cell blink for DataGridView

2008-01-11T08:28:00.000-08:00

After reading many articles on Code project, I realized that it's my time to contribute. Few months ago, I came across a requirement for cells in a DataGridView control to blink when the cell value changed. The code presented here can be applied to any other grid.

The blinking of the grid cell is achieved in the following manner. When we update the value of a cell, we also change the background color of that cell to a blink color. To restore the cell background color to its original value, we run a background thread that iterates through a list of cells that are blinking and resets them to their original non blinking state.

The code
The sample project has two functions. The first function DataInputThreadFunc() is used to generate random values to be filled / updated in the grid. The second function GridBlinkThreadFunc() is used to restore the cells to the non blink state.

Let's take a look at the first function DataInputThreadFunc():

Collapseprivate void DataInputThreadFunc()
{
Random rand = new Random();
while (true)
{
if (dataGridView1.IsDisposed)
break;

CellData data = new CellData();
data.Row = rand.Next(0, 7);
data.Col = rand.Next(0, 3);
data.Time = DateTime.Now;

int value = rand.Next(0, 101);

dataGridView1.Invoke((MethodInvoker)delegate()
{
dataGridView1.Rows[data.Row].Cells[data.Col].Value = value;
dataGridView1.Rows[data.Row].Cells[data.Col].Style
.BackColor = Color.Salmon;
});

lock (_blinkData)
{
_blinkData.Add(data);
}

Thread.Sleep(1000);
}
}

The function uses a while (true) loop as it's a background thread and will be shutdown automatically when the application is closed. if (dataGridView1.IsDisposed) check is done to make sure we do not call dataGridView1.Invoke() on a disposed object. This can happen when the user closes the application.

Next, we initialize an object of the class CellData to store the blink data.

class CellData
{
public int Row;
public int Col;
public DateTime Time;
}

This class is used to store the row number, column number and the time when the value changed.

Next we use dataGridView1.Invoke() to make a call to the user interface thread and set the grid properties. We save the blink data in a generic list to be used later by the blink thread function. Since the list is altered by more than one thread, we synchronize access by locking the list on each access.

Now let's take a look at the blink thread function.

Collapseprivate void GridBlinkThreadFunc()
{
while (true)
{
// Make a copy to avoid invalid operation exception
// while iterating through the map
List tempBlinkData;
lock (_blinkData)
{
tempBlinkData = new List(_blinkData);
}

foreach (CellData data in tempBlinkData)
{
TimeSpan elapsed = DateTime.Now - data.Time;
if (elapsed.TotalMilliseconds > 500) // 500 is the Blink delay
{
if (dataGridView1.IsDisposed)
return;

dataGridView1.BeginInvoke((MethodInvoker)delegate()
{
dataGridView1.Rows[data.Row].Cells[data.Col]
.Style.BackColor = dataGridView1.Columns[data.Col]
.DefaultCellStyle.BackColor;
});

lock (_blinkData)
{
_blinkData.Remove(data);
}
}
}

Thread.Sleep(250); // Blink frequency
}
}

At the very beginning, we make a copy of the _blinkData list. This helps us to modify the list while we iterate through the contents of the temporary copy. For each cell we find in the list, we check to make sure whether the blink time has elapsed or not. In this case the blink time is 500 milliseconds. Any cell that has elapsed the blink time gets its background color reset to the default cell style background color and is removed from the list.

Again we make sure that we set the grid property only in the user interface thread. In addition, we lock the _blinkData list before altering it. Thread.Sleep(250) is the frequency with which we go through the list to turn off the cells. Ideally, it should be half the value of blink delay.

Points of Interest
You will notice this code can be applied to any grid. This code can also be hidden in a class that extends a DataGridView control.

One thing I love about .NET 2.0 is this dataGridView1.Invoke((MethodInvoker)delegate(). This statement lets you get away from writing a function and declaring a delegate. That frees up tons of browsing time!

A good point was made by "Kristof Verbiest" about the use of BeginInvoke() instead of Invoke(). The GridBlinkThreadFunc() uses BeginInvoke() to avoid unnecessary context switch.

A Class to Save and Load Listbox Data

2008-01-11T08:27:00.000-08:00

Introduction
This is my first attempt at writing a class and at writing a Code Project article so I hope it will be of some use. I noticed I was using a lot of listbox saving/restoring in my applications so I decided to write a class for this. The class does not contain error handling but it should work fine. To use it just add the files to your project and include ListBoxSafe.h. There are (only) two functions you can use (simplicity):

void LoadList(CListBox &list, CString FileName, int MaxItems, int ItemLen);

and

void SaveList(CListBox &ListToFill, CString FileName, int MaxItems, int MaxItemLen);

list/ListToFill is the ListBox that you need to save or fill.
FileName is the name of the file from which you want to load or to which you want to save to.
MaxItems is the maximum amount of items to be saved or loaded.
ItemLen/MaxItemLen is the maximum length of a listbox item (should be the same if you save and load).

You can give MaxItems the value 0, and it will be 999 by default.
You can give MaxItemLen/ItemLen the value 0, and it will be 30 by default.

Example
CListBoxSafe::SaveList(m_lMyList,"test.dat",0,0);
CListBoxSafe::LoadList(m_lDataList,"data.txt",0,0);

Using virtual lists

2008-01-11T08:26:00.000-08:00

Introduction
Let's say you have a large database in your program and you want to show the database to the user. You use a CListCtrl with a couple of columns and fill it with a few thousand, maybe million elements. When you run it, you notice it's a bit (or very) slow. Wouldn't it be great if you didn't need to add all elements to the list, and let the list show them anyway? Does it sound stupid and ridiculous? That's how a virtual list works.

Virtual list
A virtual list is a list that has no data, it only knows how many data items it is supposed to have. But how does it now what data to show? The secret is that the list asks the parent for the information it needs. Assume you have a list with 100 elements, and elements 10-20 are visible. When the list is redrawing, it first asks the parent about element 10. When the parent has answered, the list redraws element 10, and then goes on to the next element.

A virtual list is sending three different messages to the parent. LVN_GETDISPINFO is sent when the list needs information. This is the most important message. The message LVN_ODFINDITEM is sent when the user tries to find an item by writing in the list. LVN_ODCACHEHINT is sent to give you a chance to cache data. You will probably don't care about this message at all.

OK, that's enough fuzzy theory, let's look on some code instead.

Creating a virtual list
Creating a virtual list isn't much harder than creating an ordinary CListCtrl. Add a list control in the resource editor as you usually do. Then check the style "Owner data", and then add a CListCtrl variable for this control. The only difference from an ordinary CListCtrl is the "Owner data" (LVS_OWNERDATA) style.

When you work with a virtual list, you use it mostly in the same way you do with a non-virtual list. Adding columns, selecting items, adding image list and much more works exactly the same way.

Add items to the list
Let's say m_list is the control variable for the list. Normally, you add data to the list like this:

m_list.InsertItem(0, _T("Hello world"));

But in a virtual list, this will not work. Instead, it is up to you to handle the data. Instead of adding, you change the number of elements the list is showing:

//"Add" 100 elements
m_list.SetItemCount(100);

If you set the item count to 100 or 1,000,000 doesn't matter, the time to run this command will still be practically zero. In a non-virtual list, adding a million elements could take hours.

Handling the LVN_GETDISPINFO message
As I said before, the list is asking the parent for information when it needs it. The list does this by sending a LVN_GETDISPINFO message. This is the most important message to handle when you are dealing with a virtual list. A typical function looks like this:

Collapsevoid CVirtualListDlg::OnGetdispinfoList(NMHDR* pNMHDR, LRESULT* pResult)
{
LV_DISPINFO* pDispInfo = (LV_DISPINFO*)pNMHDR;

//Create a pointer to the item
LV_ITEM* pItem= &(pDispInfo)->item;

//Which item number?
int itemid = pItem->iItem;

//Do the list need text information?
if (pItem->mask & LVIF_TEXT)
{
CString text;

//Which column?
if(pItem->iSubItem == 0)
{
//Text is name
text = m_database[itemid].m_name;
}
else if (pItem->iSubItem == 1)
{
//Text is slogan
text = m_database[itemid].m_slogan;
}

//Copy the text to the LV_ITEM structure
//Maximum number of characters is in pItem->cchTextMax
lstrcpyn(pItem->pszText, text, pItem->cchTextMax);
}

//Do the list need image information?
if( pItem->mask & LVIF_IMAGE)
{
//Set which image to use
pItem->iImage=m_database[itemid].m_image;

//Show check box?
if(IsCheckBoxesVisible())
{
//To enable check box, we have to enable state mask...
pItem->mask |= LVIF_STATE;
pItem->stateMask = LVIS_STATEIMAGEMASK;

if(m_database[itemid].m_checked)
{
//Turn check box on
pItem->state = INDEXTOSTATEIMAGEMASK(2);
}
else
{
//Turn check box off
pItem->state = INDEXTOSTATEIMAGEMASK(1);
}
}
}

*pResult = 0;
}

First, we create a LV_DISPINFO pointer, and then a pointer to the item. In itemid, we save which item we should handle. Then we check the mask in pItem. The mask is telling us what kind of information the list needs. First, we check if text information is needed. If it is, we first figure out which column the text is. The first column is for names, the second for slogans (several columns will be shown in report view, in other views only first column will be used).

We also check if image information is needed. If it is, we save which image to use in pItem->iImage (this is the number to an image in the image list connected to the list).

If check boxes are visible, we should send information about that when image information is requested. First, we change pItem->mask to tell that we are sending state information. We also change pItem->stateMask to tell what kind of information we are sending. Then we write if check box is on or off.

The mask could also have the flags LVIF_INDENT, LVIF_NORECOMPUTE, LVIF_PARAM, and LVIF_DI_SETITEM. But I have never used any of them, so I guess they aren't important :-).

It isn't harder to handle the LVN_GETDISPINFO message than this. Check boxes are probably the hardest, but you will probably never use any more complicated code than I have done in this example. When you have written this function, your list will almost act like a normal list. However, it might be a good idea to implement the LVN_ODFINDITEM as well.

Handling the LVN_ODFINDITEM message
First, some basic education: start Explorer and go to a folder where you have a lot of files. Press A. What happens? If you have a file or folder that begins with an 'A', that file should now be selected. Press A again. If you have more than one file that begins with an 'A', the second file should be selected. Write "AB". If there is any file that begins with 'AB', that is now selected. This is how every normal list control behaves. Aren't list controls cool? :-).

Let's look on how a list control normally searches:

Name
Anders
Anna
Annica
Bob
Emma
Emmanuel

Anna is selected. When we are writing anything, the list will search down to find the best match. If it reaches the end, it restarts at the top and searches until it is back on the start item (Anna). If "A" is written, Annika should be selected. If "AND" is written, Anders should be selected. If "ANNK" is written, the selection should stay on Anna. If "E" is written, Emma should be selected.

Unfortunately, this doesn't work with virtual lists. A virtual list doesn't try to find any item at all, unless you handle the LVN_ODFINDITEM message. I usually implement the message like this:

Collapsevoid CVirtualListDlg::OnOdfinditemList(NMHDR* pNMHDR, LRESULT* pResult)
{
// pNMHDR has information about the item we should find
// In pResult we should save which item that should be selected
NMLVFINDITEM* pFindInfo = (NMLVFINDITEM*)pNMHDR;

/* pFindInfo->iStart is from which item we should search.
We search to bottom, and then restart at top and will stop
at pFindInfo->iStart, unless we find an item that match
*/

// Set the default return value to -1
// That means we didn't find any match.
*pResult = -1;

//Is search NOT based on string?
if( (pFindInfo->lvfi.flags & LVFI_STRING) == 0 )
{
//This will probably never happend...
return;
}

//This is the string we search for
CString searchstr = pFindInfo->lvfi.psz;

int startPos = pFindInfo->iStart;
//Is startPos outside the list (happens if last item is selected)
if(startPos >= m_list.GetItemCount())
startPos = 0;

int currentPos=startPos;

//Let's search...
do
{
//Do this word begins with all characters in searchstr?
if( _tcsnicmp(m_database[currentPos].m_name,
searchstr, searchstr.GetLength()) == 0)
{
//Select this item and stop search.
*pResult = currentPos;
break;
}

//Go to next item
currentPos++;

//Need to restart at top?
if(currentPos >= m_list.GetItemCount())
currentPos = 0;

//Stop if back to start
}while(currentPos != startPos);
}

It may not be obvious how this works at a first look, but if you read it carefully, you will understand. Or, you simply skip this, and copy the code and just do the necessary changes - it's up to you :-). If the list is really large, maybe you need to make a faster version of this function, or don't implement it at all.

pFindInfo->lvfi has information on how you should search (like "Restart at top if bottom is reached" or in which direction). I have never cared about that, if you do, you should read in MSDN to get more information.

Handling the LVN_ODCACHEHINT message
LVN_ODCACHEHINT is sent to give you a chance to cache data. If you are working with a database that is in another computer in some network, maybe this is useful, but I haven't used this message in any of my programs. A function that handles this message will probably look something like this:

void CVirtualListDlg::OnOdcachehintList(NMHDR* pNMHDR, LRESULT* pResult)
{
NMLVCACHEHINT* pCacheHint = (NMLVCACHEHINT*)pNMHDR;

// ... Cache the data pCacheHint->iFrom to pCacheHint->iTo ...

*pResult = 0;
}

This is quite simple as you see. But as usual, simple things don't work :-). According to MSDN, you should override OnChildNotify and add the handler in this function. But I will not dig deeper in this, if you need more information about this, read in MSDN.

Changing an item
What should you do to change the data? This is really simple. You don't change the data in the list, but in the database instead. To redraw the list items, call CListCtrl::RedrawItems.

Check boxes
Check boxes are useful, but they are quite tricky to implement when you are working with a virtual list. In a normal non-virtual list, check boxes are toggled when you click on them or when you press space. But in a virtual list, nothing will happen. So you have to implement these events yourself. We start with a simple toggle-check-box-function:

void CVirtualListDlg::ToggleCheckBox(int item)
{
//Change check box
m_database[item].m_checked = !m_database[item].m_checked;

//And redraw
m_list.RedrawItems(item, item);
}

We will call this function when we want to change an item. The function toggles the check box value (in the database!) and forces the list to redraw the item. Quite simple. Toggling a check box when space is pressed is also quite simple. Add a message handler for the LVN_KEYDOWN message. The function should be something like this:

void CVirtualListDlg::OnKeydownList(NMHDR* pNMHDR, LRESULT* pResult)
{
LV_KEYDOWN* pLVKeyDown = (LV_KEYDOWN*)pNMHDR;

//If user press space, toggle flag on selected item
if( pLVKeyDown->wVKey == VK_SPACE )
{
//Toggle if some item is selected
if(m_list.GetSelectionMark() != -1)
ToggleCheckBox( m_list.GetSelectionMark() );
}

*pResult = 0;
}

We check if space is pressed and if any item is selected before we toggle the check box. To toggle check box when we click on it, we have to do a more complicated function. Add a message handler for the NM_CLICK message. My function looks like this:

void CVirtualListDlg::OnClickList(NMHDR* pNMHDR, LRESULT* pResult)
{
NMLISTVIEW* pNMListView = (NM_LISTVIEW*)pNMHDR;

LVHITTESTINFO hitinfo;
//Copy click point
hitinfo.pt = pNMListView->ptAction;

//Make the hit test...
int item = m_list.HitTest(&hitinfo);

if(item != -1)
{
//We hit one item... did we hit state image (check box)?
//This test only works if we are in list or report mode.
if( (hitinfo.flags & LVHT_ONITEMSTATEICON) != 0)
{
ToggleCheckBox(item);
}
}

*pResult = 0;
}

We are using CListCtrl::HitTest to see if we clicked on an item. If we did, the function returns which item we clicked on. We then use hitinfo.flags to see where we clicked. If the flag LVHT_ONITEMSTATEICON is on, then we know that the user clicked on the check box (state image).

Unfortunately, CListCtrl::HitTest doesn't work as it should if the list view is in "Icon" or "Small icon" mode. In these views, hitinfo.flags & LVHT_ONITEMSTATEICON is always 0. I haven't found a solution to this, if you do, let me now. However, using check boxes in these modes is probably quite unusual, so this is not a big problem.

Notes
Unless you are making something very unusual, you will not need to know more information than there is in this article, to use virtual lists. However, there are some minor compatibility issues between a virtual and a non-virtual list. For example, a virtual list can't sort data. But that is quite obvious, isn't it :-)? You find more information about these issues in MSDN.

When should you use a virtual list, and when should you not? For large lists, I personally prefer a virtual list. But a non-virtual list is usually easier to program (message handling is never fun), so for smaller lists, I use the ordinary list.

A very neat thing about virtual lists is that they are very easy to keep synchronized with a database. You just change the database and redraw the list if necessary. So, if you work with a list where the user should change data in the database, a virtual list could be useful even if number of items are low.

Another nice thing is that you sometimes could generate data when necessary. If you want to show row number in one column, that is very easy to do, and it takes practically no memory at all. In a non-virtual list, you would have to add this data on all items.

History

Clipboard backup (Visual C++)

2008-01-11T08:25:00.001-08:00

Introduction
This piece of code is to help you backup your clipboard data and restore it after you have finished other clipboard operations.

My experience: while I'm developing Word add-in programs, the icon of the new-added button can only be set through clipboard by "...->PasteFace()". This will empty the clipboard data which is going to be pasted in Word. So, I wrote this class, and it helped me a lot.

Usage:

// the constructor will do backup clipboard operation
CClipboardBackup cbbackup;

// any other clipboard operations
::OpenClipboard(NULL);
::EmptyClipboard();
::SetClipboardData(......);
::CloseClipboard();
....

// restore
cbbackup.Restore();

Advertisement

Copying a Bitmap of the Client Area

2008-01-11T08:23:00.000-08:00

The following bit of code renders the client area using OnDraw() into a bitmap. The bitmap is placed on the clipboard as a CF_BITMAP that is recognizable by most applications that accept bitmaps. This code will work will all mapping modes provided that mapping mode is set in the OnPrepareDC() function.

Collapsevoid CXStitchView::OnEditCopy()
{
CRect rect;
CClientDC dc(this);
CDC memDC;
CBitmap bitmap;

GetClientRect(&rect);

// Create memDC
memDC.CreateCompatibleDC(&dc);
bitmap.CreateCompatibleBitmap(&dc, rect.Width(), rect.Height());
CBitmap* pOldBitmap = memDC.SelectObject(&bitmap);

// Fill in memDC
memDC.FillSolidRect(rect, dc.GetBkColor());
OnPrepareDC(&memDC);
OnDraw(&memDC);

// Copy contents of memDC to clipboard
OpenClipboard();
EmptyClipboard();
SetClipboardData(CF_BITMAP, bitmap.GetSafeHandle());
CloseClipboard();

// Clean up
memDC.SelectObject(pOldBitmap);
bitmap.Detach();
}

Copying a Table of Data to the Clipboard
Placing a table of data on the clipboard is easy. It is simply a string of text. Tabs separate columns, new lines separate rows. Here's some example source.

#include

void CClipExamView::OnEditCopy()
{
// Create a shared memory file
CSharedFile sf(GMEM_MOVEABLE | GMEM_SHARE | GMEM_ZEROINIT);

// Place clipboard data in the shared memory file
RenderTableData(sf);

if (sf.GetLength() > 0) {
// Put the data on the clipboard
OpenClipboard();
EmptyClipboard();
SetClipboardData(CF_TEXT, sf.Detach());
CloseClipboard();
}
}

void CClipExamView::RenderTableData(CFile &file)
{
// Columns are separated by tabs ('\t')
// Rows are separated by new lines ('\n')
CString buffer = "1\t2\t3\n4\t5\t6\n";
file.Write(buffer, buffer.GetLength());
}

Copying Formatted Data to the Clipboard
Formatted text can support bold, italic or any other formatting that can be done in a word processor. Formatted text is usually placed on the clipboard as RTF (Rich Text Format).

The Rich Text Format is intended as an interchange format for Word-processing applications. Because of that, it is a rather large and feature rich file format. Fortunately, it is possible to describe a minimal RTF command set for creating simple formatted documents.

Basic RTF commands:
\par - Starts a new paragraph.
\tab - A tab.
\b - Enable Bold (scoped within a group)
\i - Enable Italics (scoped within a group)
\ul - Enable Underline (scoped within a group)

For example, the RTF string:

{\rtf1 {1 \tab 2 \tab 3 \par 4 \tab {\b\i 5} \tab 6}}

Source

#include

void CClipExamView::OnEditCopy()
{
// Create a shared memory file
CSharedFile sf(GMEM_MOVEABLE | GMEM_SHARE | GMEM_ZEROINIT);

// Place clipboard data in the shared memory file
RenderFormattedData(sf);

if (sf.GetLength() > 0) {
// Put the data on the clipboard
OpenClipboard();
EmptyClipboard();
SetClipboardData(::RegisterClipboardFormat(CF_RTF), sf.Detach());
CloseClipboard();
}
}

void CClipExamView::RenderFormattedData(CFile &file)
{
// RTF (Rich Text Format) - Don't forget to escape
// the \ character in your C strings!
CString buffer = "{\\rtf1 {1 \\tab 2 \\tab 3"
" \\par 4 \\tab {\\b\\i 5} \\tab 6}}";
file.Write(buffer, buffer.GetLength());
}

Getting More Info on RTF

Data Drag is coming to town... the drop target

2008-01-11T08:20:00.000-08:00

You imagine? Right, there's more than one possibility, this time I'll give you tree. One for your private data, one for the common data in order to receive data from other applications like Excel, WinWord etc. and at last, I'll give you a handy-dandy class you can derive ANY MFC object from, to make it a drop target.

Getting the data from the dropped object, the MFC way
I assume you want to drag the contents of some rows of a CListControl of your CFormView into one of the modal dialogs mentioned earlier in this article series. In real-life, you can use this approach to implement the drop functions to any CWnd-derived object, whether in the modal dialog or not.

The first steps are the same as done when implementing the drop source:

To your drop target class, e.g. my CDialog based class CDropDialog, add a member variable of type UINT: UINT m_DragDropFormat;

In the constructor of the view, initialize it for private data exchange with: m_DragDropFormat =
::RegisterClipboardFormat("YourInterfaceClipboardFormat");

and for the common format just use:

m_DragDorpFormat = CF_TEXT;

Add the following lines to stdafx.h: #include
#include

In your application's InitInstance function, make a call to AfxOleInit() before creating the document template
If the window used as drop target is CView-derived, you are now in luck, most of it is at the hand. In the OnInitialUpdate function, call the Register() function and overwriting the functions OnDragOver and OnDrop (implementation same as CInterfaceDropTarget in step 9) should be enough in "normal" cases - READY!
If you have another kind of target, there's some more work.
Use the ClassWizard to create a new generic(!) class derived from the MFC(!) class COleDropTarget. I call it CInterfaceDropTarget.
Create and initialize a UINT m_DragDropFormat as in step 2.
Create - at least - the functions OnDragOver and OnDrop. CollapseDROPEFFECT CInterfaceDropTarget::OnDragOver (CWnd *pDrop,
COleDataObject* pDataObject, DWORD dwKeyState, CPoint point )
{
CFile *pFile = pDataObject->GetFileData(m_DragDropFormat);
if (pFile != NULL)
// perhaps some point checking here?
return DROPEFFECT_COPY; // data fits
else
return DROPEFFECT_NONE; // data won't fit
}

BOOL CInterfaceDropTarget::OnDrop(CWnd *pDrop,
COleDataObject* pDataObject, DROPEFFECT dropEffect, CPoint point )
{
CFile *pFile = pDataObject->GetFileData(m_DragDropFormat);
if (pFile != NULL)
{
int number, colCou;
CString Data;
// perhaps some point checking first?
TRY
{
CArchive ar(&file, CArchive::load);
TRY
{
if (DragDropFormat != CF_TEXT)
{
// "Serialize" your data object from the archive
// (yes, you may use YourObject.Serialize(ar) here!)
ar >> number;
ar >> colCou;

for (int i=0; i {
ar >> Data ;
#pragma message(__FILE__ " dont't forget to process Data here.....")
}
}
else
{
CString Data, t;

TRY
{
while(ar.ReadString(t))
Data += t + "\n";
}
#pragma message(__FILE__ " ... and here!")
CATCH_ALL(eInnerMost)
{
#ifdef _DEBUG
TCHAR szCause[255];
CString strFormatted;
eInnerMost->GetErrorMessage(szCause, 255);
strFormatted = _T("Exception: ");
strFormatted += szCause;
AfxMessageBox(strFormatted);
#endif //_DEBUG
ASSERT(FALSE);
}
END_CATCH_ALL;
}
ar.Close();
}
CATCH_ALL(eInner)
{
// exception while reading
// from or closing the archive
ASSERT(FALSE);
}
END_CATCH_ALL;
}
CATCH_ALL(eOuter)
{
// exception in the destructor of ar
ASSERT(FALSE);
}
END_CATCH_ALL;
return TRUE;
}
return COleDropTarget::OnDrop(pDrop,
pDataObject, dropEffect, point);
}

Don't forget to process the data in the OnDrop function. You should send the just-dropped-object to your CWnd-class and "do something" with it there. The example does this with the functions ProcessData(CString data) and ProcessData(int rowcount, int linecount, CString Data).
In the CWnd-derived class used as drop target, add a member variable of type CInterfaceDropTarget, for example's sake name it m_DT.
At any point where this window exists (for example: OnInitDialog is great) call the Register function of m_DT: BOOL CDropDialog::OnInitDialog()
{
CDialog::OnInitDialog();
m_DT.Register(this);
// the other stuff follows ....

Time Management for System Administrators

2008-01-11T06:16:00.000-08:00

Time Management for System Administrators contains useful advice for SysAdmins looking to use their time more effectively. If you are a techie who wonders why you get bogged down with little tasks while not getting your projects done, this is the book for you. If you don't have the time to read this book, you definitely need to read it.

--------------------------------------------------------------------------------
Copyright notice: All reader-contributed material on freshmeat.net is the property and responsibility of its author; for reprint rights, please contact the author directly.
--------------------------------------------------------------------------------

Title: Time Management for System Administrators
Author: Thomas A. Limoncelli
Publisher: O'Reilly
Purchase URL: http://www.oreilly.com/catalog/timemgmt/

This is a compact book, weighing in at around 200 pages. The size is appropriate for a book intended to help people with too little time. It is basically a pile of advice for those of us who are constantly fixing broken systems, answering user requests, attending meetings, and maybe sometimes actually working on our projects. Some of this advice you have probably figured out on your own, some you will find useful, and some you will disagree with. That's OK; take what you need and leave the rest.

The first chapter is in some ways the most valuable part of the book. It covers how to deal with interruptions, and lists some general principles of time management.

The book moves on to discuss various ways to maintain focus while working on tasks and projects, so that you actually get things done instead of responding to interruptions and destroying your concentration through multitasking. Then it recommends routines, so that you don't forget to do things and so that you don't need to worry about what to do. I have found the advice in these chapters to be useful.

The core of the book is "The Cycle" -- a system for managing your to-do items, appointments, calendar, and even long-term goals. I've been trying out parts of it, and have found that it has helped me avoid schedule conflicts, made me less likely to drop to-do items, and improved my follow-up. So far, it hasn't actually helped me get down to 40 work hours a week, although in all fairness, I'm not yet doing the full system.

The rest of the book discusses prioritization, email management (in particular, the principle of "touch once" -- which I have found to be an extremely effective way of dealing with email), time wasters, documentation (which is time management -- documentation helps you do things faster or delegate them to other people), and automation. Most of the advice here has been useful as well. My only complaint is that the discussion on shell scripting did not seem very useful -- if I didn't already know shell scripting, it would not have taught me much, or convinced me that I needed this skill for time management.

The book also contains advice on various topics that are somewhat tangential. Such topics include customer management, long-term goals, "managing your boss", stress management (which gets its own chapter), and personal time. While not directly related to the nominal purpose of the book, the points mostly seemed valid and useful.

On the whole, I'd say that this book is well worth its $25 asking price. I've known for a while that I have time management problems, so I once bought a different time management book, but never finished it; the other book seemed more oriented towards executives and managers. This book speaks to system administrators, and it does a good job. It's not a panacea -- to fix your time management problems, you not only need to know what the problems are, you also need to go to considerable effort to change your habits. But this book's advice has already helped me to improve my time management, and hopefully it will help you, too.

The Test/Code Cycle in XP: Part 1, Model

2008-01-11T06:15:00.001-08:00

This paper demonstrates the development of a small bibliographic system using Extreme Programming techniques. Part 1 shows development of the model; part 2 (only available in Java as at Dec 2000) shows development of the associated user interface.

Specifically, it shows how unit tests and simple design work together while programming. Observe how the coding process occurs in small steps, just enough implementation to make each test run. There's a rhythm to it, like a pendulum of a clock: test a little, code a little, test a little, code a little. (To bring this out, we'll align test code to the left of the page, and application code to the right.)

For this example, suppose we have bibliographic data with author, title, and year of publication. Our goal is to write a system that can search that information for values we specify. We have in mind an interface something like this:

We'll divide our work into two parts: the model and the user interface.

For our model, we have a collection of Documents, which know their metadata. A Searcher knows how to find Documents: given a Query, it will return a Result (a set of Documents). We'll create our unit tests (and our classes) bottom-up: Document, Result, Query, and Searcher.

Document, Result, and Query
Document
A Document needs to know its author, title, and year. We'll start with a "data bag" classs, beginning with its test:

public void TestDocument() {
Document d = new Document("a", "t", "y");
AssertEquals("a", d.Author);
AssertEquals("t", d.Title);
AssertEquals("y", d.Year);
}

This test doesn't compile (as we haven't created the Document class yet). Create the class with stubbed-out methods.

Run the test again to make sure it fails. This may seem funny - don't we want the tests to pass? Yes, we do. But by seeing them fail first, we get some assurance that the test is valid. And once in a while, a test passes unexpectedly: "that's interesting!"

Fill in the constructor and the methods to make the test pass.

Let's highlight this mini-process:

The Test/Code Cycle in XP
Write one test.
Compile the test. It should fail, as you haven't implemented anything yet.
Implement just enough to compile. (Refactor first if necessary.)
Run the test and see it fail.
Implement just enough to make the test pass.
Run the test and see it pass.
Refactor for clarity and "once and only once".
Repeat from the top.

This process ensures that you've seen the test both fail and pass, which gives you assurance that the test did test something, that your change made a difference, and that you've added valued functionality.

Some people will advocate not bothering to test simple properties. ("They can't possibly be wrong.", "It's a pain to write a bunch of property tests.") I tend to write the tests anyway:

It doesn't take that much time to write the test, and it's certainly not hard, but it gives you that extra edge. ("You thought you were sure it's ok; now you have a test that demonstrates it.")
A test will often have a longer lifetime than the code it's testing. The test is there so when you add caching, or don't create objects until required, or add logging, etc., you still have assurance that the original function will work.
Boring tests full of setters and getters are often trying to tell you something: the class may not be pulling its weight. When a class is almost a "struct", it's often a sign that the responsibilities aren't distributed right between classes.
Result
A Result needs to know two things: the total number of items, and the list of Documents it contains. First we'll test that an empty result has no items.

public void TestEmptyResult() {
Result r = new Result();
Assert ("count=0 for empty result", r.Count == 0);
}

Create the Result class and stub out its Count property. See it fail until you add "return 0;" as its implementation.

Next test a result with two documents.

public void testResultWithTwoDocuments() {
Document d1 = new Document("a1", "t1", "y1");
Document d2 = new Document("a2", "t2", "y2");
Result r = new Result(new Document[]{d1, d2});
Assert (r.Count == 2);
Assert (r[0] == d1);
Assert (r[1] == d2);
}
Add the indexer declaration (returning null) and watch the test fail. (I'm going to stop mentioning that, but keep doing it. It takes a few seconds, but gives that extra bit of reassurance.) Implementing a simple version of Result will give:

public class Result
{
Document[] myDocuments = new Document[0];

public Result() {}

public Result(Document[] aDocumentArray) {
myDocuments = aDocumentArray;
}

public Document this [int anIndex] {
get { return myDocuments[anIndex]; }
}

public int Count {
get { return myDocuments.Length; }
}
}
The test runs, so we're done with this class.

Query
We'll represent the Query as just its query string.

public void TestSimpleQuery()
{
Query q = new Query("test");
AssertEquals("test", q.Value);
}

Create the Query class with a constructor, so that it remembers its query string and reports it via Value.

Searcher
The Searcher is the most interesting class. The easy case is first: we should get nothing back from an empty collection of Documents.

public void TestEmptyCollection() {
Searcher searcher = new Searcher();
Result r = searcher.Find(new Query("any"));
Assert(r.Count == 0);
}

This test doesn't compile, so stub out the Searcher class.

public class Searcher {
public Searcher() {}
public Result Find(Query aQuery) { return null; }
}

The test compiles, but fails to run correctly (because Find() returns null). We can fix this with this change:

public Result Find(Query aQuery) {return new Result();}
Things get more interesting when we try real searches. Then we face the issue of where the Searcher gets its documents. We'll begin by passing an array of Documents to the Searcher's constructor. But first, a test.

public void TestOneElementCollection()
{
Document d = new Document("a", "a word here", "y");
Searcher searcher = new Searcher(new Document[]{d});

Query q1 = new Query("word");
Result r1 = searcher.Find(q1);
Assert(r1.Count == 1);

Query q2 = new Query("notThere");
Result r2 = searcher.Find(q2);
Assert (r2.Count == 0);
}

This test shows us that we have to find what is there, and not find what's not there.

To implement this, we have to provide the new constructor that makes the test compile (though it still fails). Then we have to get serious about implementation.

First, we can see that a search has to retain knowledge of its collection between calls to Find(), so we'll add a member variable to keep track, and have the constructor remember its argument:

Document[] myDocuments = new Document[0];

public Searcher(Document[] aDocumentArray) {
myDocuments = aDocumentArray;
}

Now, the simplest version of Find() can iterate through its documents, adding each one that matches the query to a Result:

public Result Find(Query aQuery) {
Result result = new Result();

foreach (Document doc in myDocuments) {
if (doc.Matches(aQuery)) {
result.Add(doc);
}
}

return result;
}

This looks good, except for two problems: Document has no Matches() method, and Result has no Add() method.

Let's add a test: we'll check that each field can be matched, and that a document doesn't match queries it shouldn't:

public void TestDocumentMatchingQuery() {
Document d = new Document("1a", "t2t", "y3");
Assert(d.Matches(new Query("1")));
Assert(d.Matches(new Query("2")));
Assert(d.Matches(new Query("3")));
Assert(!d.Matches(new Query("4")));
}

There are three situations for queries that we should deal with eventually: empty queries, partial matches, and case sensitivity. For now, we'll assume empty strings and partial matches should match, and the search is case-sensitive. In the future we might change our mind.

This is enough information to let us implement matches:

public bool Matches(Query aQuery) {
String query = aQuery.Value;

return myAuthor.IndexOf(query) != -1
|| myTitle.IndexOf(query) != -1
|| myYear.IndexOf(query) != -1;
}

This will enable TestDocumentMatchingQuery() to work, but TestOneElementCollection() will still fail because Result has no Add() method yet. So, add a test for the method Result.Add():

public void TestAddingToResult()
{
Document d1 = new Document("a1", "t1", "y1");
Document d2 = new Document("a2", "t2", "y2");

Result r = new Result();
r.Add(d1);
r.Add(d2);

Assert ("2 items in result", r.Count == 2);
Assert ("First item", r[0] == d1);
Assert ("Second item", r[1] == d2);
}

This test fails. Result already remembers its list by using an array, but that is not the best choice for a structure that needs to change its size. We'll change to use an ArrayList:

ArrayList myDocuments = new ArrayList();

public Result(Document[] aDocumentArray) {
foreach (Document doc in aDocumentArray) {
myDocuments.Add(doc);
}
}

public Document this [int anIndex] {
get { return (Document)myDocuments[anIndex]; }
}

public int Count {
get { return myDocuments.Count; }
}

Make sure the old unit tests TestEmptyResult() and TestResultWithTwoDocuments() still pass. Add the new method:

public void Add(Document aDocument)
{
myDocuments.Add(aDocument);
}

Let's consider the the new Result(Document[]) constructor. It was introduced to support the TestResultWithTwoDocuments() test, because it was the only way we could create Results containing documents. Later, we introduced Result.Add(), which is what Searcher needs. The array constructor is no longer needed. So, we'll put on a testing hat and revise that test. Instead of Result r = new Result(new Document[]{d1,d2}), we'll use:

Result r = new Result();
r.Add(d1);
r.Add(d2);

We verify that all tests still pass, so it is now safe to remove the array-based constructor. We also see that TestAddingToResult() is now essentially a duplicate of TestResultWithTwoDocuments(), so we'll remove the latter.

Finally, all our tests pass for Document, Result, Query, and Searcher.

Initialization
Loading Documents
Where does a searcher get its documents? Currently, you'd call its constructor from the main routine, passing in an array of documents. Instead, we want the searcher to own the process of loading its documents.

We begin with a test. We'll pass in a Reader, and be prepared to see exceptions. We've also postulated a Count property, used only by tests to verify that something was loaded. An advantage of having the tests in the same Assembly as the class under test is that you can provide non-public methods that let tests view an object's internal state.

public void TestLoadingSearcher() {
try {
String docs = "a1\tt1\ty1\na2\tt2\ty2"; // \t=field, \n=row
StringReader reader = new StringReader(docs);
Searcher searcher = new Searcher();
searcher.Load(reader);
Assert("Loaded", searcher.Count == 2);
}
catch (IOException e) {
Fail("Loading exception: " + e);
}
}

Notice that Searcher still uses an array (the simplest choice at the time). We'll do as we did for Result, a refactoring converting from an array to an ArrayList.

// Searcher:
ArrayList myDocuments = new ArrayList();

public Searcher() {}

public Searcher(Document[] aDocumentArray) {
foreach (Document doc in aDocumentArray) {
myDocuments.Add(doc);
}
}

public Result Find(Query aQuery) {
Result result = new Result();

foreach (Document doc in myDocuments) {
if (doc.Matches(aQuery)) {
result.Add(doc);
}
}

return result;
}

(Verify that the old tests pass.) Now we're in a position to do the loading:

// Searcher:
public Query MakeQuery(string aQueryString) {
return new Query(aQueryString);
}

public void Load(TextReader aReader) {
try {
String line = aReader.ReadLine();
while (line != null) {
myDocuments.Add(new Document(line));
line = aReader.ReadLine();
}
}
finally {
try { aReader.Close(); }
catch (Exception) { /*ignore*/ }
}
}

internal int Count {
get { return myDocuments.Count; }
}

// Document:
public Document(String line) {
string[] tokens = line.Split(new char[] {'\t'});

if (tokens.Length != 3) {
throw new ArgumentException("Author, " +
"Title and Year not provided.");
}

myAuthor = tokens[0];
myTitle = tokens[1];
myYear = tokens[2];
}

Searcher's array-based constructor is no longer needed. We'll adjust the test and delete the constructor:

public void TestOneElementCollection() {
Searcher searcher = new Searcher();

try {
StringReader reader = new StringReader("a\ta word here\ty");
searcher.Load(reader);
}
catch (Exception ex) {
Fail ("Couldn't load Searcher: " + ex);
}

Query q1 = searcher.MakeQuery("word");
Result r1 = searcher.Find(q1);
Assert(r1.Count == 1);

Query q2 = searcher.MakeQuery("notThere");
Result r2 = searcher.Find(q2);
Assert (r2.Count == 0);
}

SearcherFactory
Where does a Searcher come from? Currently, that's left up to whoever calls its constructor. Instead of letting clients depend on the constructor, we'd like to introduce a factory method responsible for locating the Searcher. (For the test, we'll put a file "test.dat" in the directory for testing. If we wanted to be less lazy, we'd have the test create and delete the file as well.) public void TestSearcherFactory() {
try {
Searcher s = SearcherFactory.GetSearcher ("test.dat");
Assert (s != null);
}
catch (Exception ex) {
Fail ("SearcherFactory can't load: " + ex);
}
}

We can implement:

public class SearcherFactory {
public static Searcher GetSearcher(String filename) {
Searcher s = new Searcher();
s.Load(new StreamReader(
new FileStream(filename, FileMode.Open)));
return s;
}
}

Now, a client obtains a Searcher by asking a SearcherFactory to give it one.

Looking Back
I'd like to put a design hat on, and look at the methods we've developed, from two perspectives: the search client and the Searcher class. Who uses each public method and property?

Search Client Searcher class
Document.Author
Document.Title
Document.Year
new Query()
Result.Count
Result[index]
Searcher.Find() new Document()
Document.Matches()
Query.Value
new Result()
Result.Add()

Looking at the Document and Query classes, I still have twinges that say they may not be doing enough (being not much more than a "data bag"). But both seem like good, meaningful "near-domain" classes, so we'll hold off on any impulse to change them. The Result and Searcher classes feel like they have the right balance.

What about the development process? It seemed to generate some blind alleys. For example, we had to change data structures from arrays to array lists (twice!). Is this a flaw in our process? No, it's not. The array was an adequate structure when it was introduced, and it was changed when necessary. We don't mind blind alleys, as long as they're never one-way dead ends. We're not omniscient, so there will be times we need to change our minds; the key is making sure we never get stuck with a bad or over-complex design.

Moving Forward: Interfaces
The implementation we've derived above is a good starting point, but is not in final form. In real systems, the bibliographic information is often kept elsewhere, perhaps in a database, XML file, on another network, etc. We don't want our clients to know which alternative they're using.

The methods in the "Search Client" column of the table above show the interfaces required by clients. "Query" is probably OK as a class (since clients have to be able to construct them), but we would like to introduce interfaces for Searcher, Result, and Document. We'll apply the "Extract Interface" refactoring (from Fowler's book).

Unfortunately, the names we'd like for our interfaces are the same as the ones we already use for the classes. Since we'd like things to be better from the client point of view, and the classes so far are based on strings, we'll rename Searcher to StringSearcher, etc. and reserve the shorter names for the interfaces.

So, move Searcher.java to StringSearcher.java. Fix every call site and reference. Run the tests to verify that we've renamed correctly.

Introduce the interface:

public interface Searcher {
Result Find(Query q);
}

(Run the tests.) Make StringSearcher implement the interface. (Run the tests.) Now, the only place that must reference StringSearcher by name is the SearcherFactory interface. (We could remove that dependency, and perhaps also put the String* objects in a different package, but we won't do that here for reasons of space.)

Apply the same process to Result, renaming the old Result to StringResult, and introducing the interface:

public interface Result {
Document this [int anIndex] { get; }
int Count { get; }
}

The StringSearcher class should still construct a StringResult object, but its return type should remain Result. (We don't mind if the String* classes depend on each other, but we don't want to make clients aware of that fact.)

Finally, introduce the interface for Document:

public interface Document {
string Author { get; }
string Title { get; }
string Year { get; }
}

We're left with two concrete classes that clients will depend on: SearcherFactory and Query. Clients depend on the interfaces for Searcher, Result, and Document, but not directly on the classes implementing those interfaces.

Conclusion
We've developed the bibliographic system's model in a typical Extreme Programming style, emphasizing simple design and a process of alternately testing and coding. The unit tests supported us in designing, coding, and refactoring. The resulting system could have either a simple command line or a graphical user interface attached to it.

Resources
Source code (zip file 4KB).
Dot Net Dan's .NET Discussion
The Test/Code Cycle in XP: Part 1, Model, in Java by William Wake
The Test/Code Cycle in XP: Part 2, GUI, in Java by William Wake.
William Wake's XPlorations
Extreme Programming Explained, Kent Beck.
Refactoring, Martin Fowler.
NUnit home
JUnit home
[Written by William Wake 1-25-2000; re-titled and revised 2-3-2000; added search.zip 7-2-00; Ported to C# by Dan Green 12-23-2000.]

Revving .NET Remoting

2008-01-11T06:10:00.000-08:00

For the first remoting app we build, everything we do will be explicit in the code. No config files or IIS setup required. The first class to look at is the Logger, itself:

namespace DotNetDan.Samples.Remoting.Log
{
using System;

public class Logger : MarshalByRefObject
{
public void Log(String aString)
{
Console.WriteLine(aString);
}
}
}

Note that the only thing we had to do to make the class remotable was implement MarshalByRefObject.

OK, let's move our attention to the server that will listen for log requests:

namespace DotNetDan.Samples.Remoting
{
using System;
using System.Runtime.Remoting;
using System.Runtime.Remoting.Channels;
using System.Runtime.Remoting.Channels.Http;
using System.Runtime.Remoting.Channels.Tcp;
using DotNetDan.Samples.Remoting.Log;

public class Server01
{
public static void Main()
{
ChannelServices.RegisterChannel(new HttpChannel(8000));
ChannelServices.RegisterChannel(new TcpChannel(8001));

RemotingConfiguration.RegisterWellKnownServiceType(
typeof(Logger), "Log", WellKnownObjectMode.Singleton);

Console.WriteLine("Log Server Listening on endpoints:\r\n" +
"\thttp://localhost:8000/Log\r\n" +
"\ttcp://localhost:8001/Log");
Console.WriteLine("Press enter to stop the server...");
Console.ReadLine();
}
}
}

Don't you love code where the using statements take up as much space as the procedural logic?
The first thing we do is register some "Channels". Channels are used by the .NET Remoting Framework to transport messages to and from objects. The client will send details relating to the method call through the channel to the remote server object. It is at the server's discretion as to what types of channels to support. In the code above we chose to register two channels to support both the TCP and HTTP transport mechanisms. These listen on ports 8000 and 8001 respectively. The HTTP channel uses the SOAP protocol; the TCP channel uses a binary protocol.

After registering the channels we register the object that will be actually listening to the server end of the channel with the RegisterWellKnownServiceType method. The parameters should be fairly self-explanatory. For a more detailed discussion refer to the online help.

When the server is now run it will wait for the enter key to be pressed. Whilst the programming is running (ie. until someone presses the enter key) it's ready to receive method calls made on the Log object.

So, let's look at the client code that will make the actual Log method calls:

namespace DotNetDan.Samples.Remoting
{
using System;
using System.Runtime.Remoting;
using DotNetDan.Samples.Remoting.Log;

public class Client01
{
public static void Main(string[] args)
{
Logger httpLogger =
(Logger)Activator.GetObject(
typeof(Logger), "http://localhost:8000/Log");
Logger tcpLogger =
(Logger)Activator.GetObject(
typeof(Logger), "tcp://localhost:8001/Log");

httpLogger.Log("HTTP Client Request");
tcpLogger.Log("TCP Client Request");
}
}
}
Since we assume the server is already running, we create the Log objects using the Server Activation Model, Activator.GetObject.

To build these examples use the following build script:

csc /t:library Log.cs
csc /r:System.Runtime.Remoting.dll /r:System.dll /r:Log.dll Server01.cs
csc /r:System.Runtime.Remoting.dll /r:system.dll ...
/r:Server01.exe /r:Log.dll Client01.cs

Using Configuration Files
In the previous example we made everything explicit so we could see what was going on. We can now make use of configuration files to make the client and server code more transparent; the code will have the appearance of everything running within the one application domain.

Let's create a server configuration file that supports both our TCP and HTTP channels:

The server code now becomes simpler, as all the imperative remoting-configuration code has been removed and replaced by a single call that points to the configuration file:

namespace DotNetDan.Samples.Remoting
{
using System;
using System.Runtime.Remoting;
using DotNetDan.Samples.Remoting.Log;

public class Server02
{
public static void Main()
{
RemotingConfiguration.Configure("server.config");

Console.WriteLine("Log Server Listening on:\r\n" +
"\thttp://localhost:8000/Log");
Console.WriteLine("Press enter to stop the server...");
Console.ReadLine();
}
}
}

Let's now look at the configuration file that supports the client:

url="http://localhost:8000/Log" />

Using this configuration file on the client allows us to write the code that has all the appearances of local instantiation. Due to the client configuration file, though, the Log object is actually invoked remotely over an HTTP channel:

namespace DotNetDan.Samples.Remoting
{
using System.Runtime.Remoting;
using DotNetDan.Samples.Remoting.Log;

public class Client02
{
public static void Main(string[] args)
{
RemotingConfiguration.Configure("client.config");

new Logger().Log("HTTP Client Request");
}
}
}

ASP.NET Auto Event Wiring

2008-01-11T06:09:00.000-08:00

we promised to discuss Page Processing. Well, that'll be the next article, we promise (again). This one's focussed on explaining some behaviour we demonstrated but couldn't immediately explain to ourselves (not that we told you:-).

Please sir, why did the Page_Load method get called?
In the previous article, we remarked that the recommended location for event handling logic is the Web Form class (aka code behind class). In all-too-typical article fashion we then went and flagrantly broke the rule, putting a Page_Load method into the .aspx file in our example, just because it made our lives easier. Well, although it did make our lives easier, it also sparked our curiosity -- why'd it work?

If we take a look at the generic Web Form class produced by Visual Studio, we can see the code that enlists event handlers (in bold):

using ...;

public class WebForm : System.Web.UI.Page {
public WebForm() {
Page.Init += new System.EventHandler(Page_Init);
}

protected void Page_Load(object sender, EventArgs e) {
if (!IsPostBack) { }
}

protected void Page_Init(object sender, EventArgs e) {
InitializeComponent();
}

private void InitializeComponent() {
this.Load += new System.EventHandler(this.Page_Load);
}
}

So just how is it that a method we create in the .aspx file -- named Page_Load -- automatically handles the load event of the Web Form without any similar binding code?

As always, the answer lies in the internals of ASP.NET. And it's pretty simple. We just have to dig around a little to find it.

ASP.NET Internals - TemplateControl's HookUp
The answer lies in the System.Web.UI.TemplateControl class (which the System.Web.UI.Page class inherits from). It contains a HookUpAutomaticHandlers method that is responsible for associating all the control events (such as init, load, databind, ...) with event handling methods that follow the naming protocol "Page_" (such as "Page_Init", "Page_Load", etc.).

Well, to be just a little more precise, the HookUpAutomaticHandlers method only does the binding if you haven't set the undocumented (as at Beta1) @Page directive called AutoEventWireup to false. Beware of this little bit of trickery because Visual Studio does set this value to false when you ask it to add a Web Form into your project. And remember, when it's set to false, "no free event handlers for you!".

Since our example contained no such AutoEventWireup attribute, the TemplateControl class automatically bound our Page_Load method to the Web Form's load event and our Button was named "FooBar" just as we hoped. Lovely.

Just as an open-ended question, why does the TemplateControl class have Page-based wiring in it when it is also the base class of UserControl? This seems a rather bad breach of The Open Closed Principle (you have read that paper like we suggested, haven't you?).

Concatenating Strings Efficiently

2008-01-10T06:26:00.000-08:00

One of the first pieces of efficiency advice most .NET developers learn is "use StringBuilder to concatenate strings". A little bit like "exceptions are expensive" this is a misunderstood piece of received wisdom. (Fortunately it's not nearly as harmful as the exception performance myth, but it comes up about as often...)

Before reading the rest of this page, you should be aware of the basics of the String type. For the sake of readability, I'll stick to "string" rather than "String" or "string" from here onwards. Please let me know if you find this confusing, and I'll change it.

I've included this in my list of general framework articles rather than in the list of C#-specific articles because I suspect every language targetting .NET is likely to use the same way of concatenating strings under the hood.

The Problem We're Trying To Avoid
There is a very real problem that this wisdom correctly addresses - that of building a load of strings (usually of increasing size) which are never used apart from contributing to the creation of other strings. Here's an example:

using System;

public class Test
{
static void Main()
{
DateTime start = DateTime.Now;
string x = "";
for (int i=0; i < 100000; i++)
{
x += "!";
}
DateTime end = DateTime.Now;
Console.WriteLine ("Time taken: {0}", end-start);
}
}

On my (fairly fast) laptop, that takes nearly 10 seconds. Double the number of iterations, and it takes over a minute. The results are slightly better on .NET 2.0 beta 2, but not hugely. The problem is that strings are immutable - just because we're using "+=" here doesn't mean the runtime actually appends to the end of the existing string. In fact, x += "!"; is absolutely equivalent to x = x+"!";. The concatenation here is creating an entirely new string, allocating enough memory for everything, copying all the data from the existing value of x and then copying the data from the string being appended ("!"). As the string grows, the amount of data it has to copy each time grows too, which is why the time taken didn't just double when I doubled the number of iterations.

This is clearly inefficient. If someone asked you to add something to a shopping list, you wouldn't write a new copy of the shopping list first, would you? Enter StringBuilder...

The StringBuilder Solution
Here is an equivalent (in terms of the final value of x) program, which is much, much faster:

using System;
using System.Text;

public class Test
{
static void Main()
{
DateTime start = DateTime.Now;
StringBuilder builder = new StringBuilder();
for (int i=0; i < 100000; i++)
{
builder.Append("!");
}
string x = builder.ToString();
DateTime end = DateTime.Now;
Console.WriteLine ("Time taken: {0}", end-start);
}
}

On my laptop, this executes too fast for the rudimentary timing mechanism I'm using to give me any sensible results. Changing to a million iterations (i.e. ten times what the first program took nearly ten seconds to do), it takes about 30-40ms. The time taken is roughly linear in the number of iterations (i.e. double the iterations and it takes twice as long). It does this by avoiding unnecessary copying - only the data we're actually appending gets copied. StringBuilder maintains an internal buffer and appends to that, only copying its buffer when there isn't room for any more data. (In fact, the internal buffer is just a string - strings are immutable from a public interface perspective, but not from within the mscorlib assembly.) We could make the above code even more efficient by passing the final size of the string (which we happen to know in this case) to the constructor of StringBuilder to make it use a buffer of the right size to start with - then there'd be no unnecessary copying at all. Unless you're in a situation where you have that information readily to hand though, it's usually not worth worrying about - StringBuilder doubles its buffer size when it runs out of room, so it doesn't end up copying the data very many times anyway.

So I Should Use StringBuilder Everywhere, Right?
No, quite simply. The above is an explanation of why the received wisdom of "use StringBuilder for concatenation" is right some of the time. However, many people take it at face value without understanding the reasoning behind it. They start turning code like this:

string name = firstName + " " + lastName;
Person person = new Person (name);

into code like this:

// Bad code! Do not use!
StringBuilder builder = new StringBuilder();
builder.Append (firstName);
builder.Append (" ");
builder.Append (lastName);
string name = builder.ToString();
Person person = new Person (name);

All that in the name of inefficiency. Now, on a broader point, even if the second version were more efficient than the first, it probably wouldn't be significantly more efficient - unless that code ended up being called vast, vast numbers of times, the amount of time spent in the concatenation is likely to be tiny. Making something less readable (and I think you'll agree the second version takes far longer to understand) for a small performance gain is a really bad idea in the first place.

However, the second version is actually less efficient than the first! Not much less efficient - and if the second version were more readable, I'd go with it for the reasons above - but when the entire point of using StringBuilder is to improve efficiency, using the above is just plain nuts.

The first version (assuming that firstName and lastName are real variables, and not constants - I'll come onto that later) compiles to a call to String.Concat, like this:

string name = String.Concat (firstName, " ", lastName);
Person person = new Person (name);

String.Concat takes a bunch of strings (or objects) and concatenates them together, plain and simple. There are various overloads - some take strings, some take objects (which are just converted into strings), some take arrays of objects or arrays of strings. They all do the same thing though. Now, String.Concat can work out the lengths of all the strings involved before it concatenates them together (at least if you pass it strings - if you pass it objects, it needs to create temporary strings and then concatenate those together). This means that no extra copying is involved - the data is copied once into the new string, which is of exactly the right length.

Compare this with the StringBuilder version. It doesn't know at construction time how big to make the buffer (because we haven't told it - doing so would make the code even less readable). That means it may have to copy the buffer, and it's likely to end up with a buffer which is actually larger than it needs to be. Oh, and there's the overhead of an extra object (the StringBuilder itself). Remind me why this was meant to be a good idea?

The important difference between this example and the previous one is that we can easily present all the strings which need to be concatenated together in one call to String.Concat. That means that no intermediate strings are needed. StringBuilder is efficient in the first example because it acts as a container for the intermediate result without having to copy that result each time - when there's no intermediate result anyway, it has no advantage.

Constants
Things get even crazier when it comes to string constants (literals, const string members). What do you suppose string x = "hello" + " " + "there"; is compiled to? It would be reasonable to expect it to be another call to String.Concat - but it isn't. It's actually compiled to the exact same code as string x = "hello there";. The compiler knows that all the parts are constant, so it does all the concatenation at compile time, storing the full string in the compiled code. Converting that to use StringBuilder is inefficient in both memory and speed, as well as reducing readability.

Rules Of Thumb
So, when should you use StringBuilder, and when should you use the string concatenation operators?

Definitely use StringBuilder when you're concatenating in a non-trivial loop - especially if you don't know for sure (at compile time) how many iterations you'll make through the loop. For example, reading a file a character at a time, building up a string as you go using the += operator is potentially performance suicide.
Definitely use the concatenation operator when you can (readably) specify everything which needs to be concatenated in one statement. (If you have an array of things to concatenate, consider calling String.Concat explicitly - or String.Join if you need a delimiter.)
Don't be afraid to break literals up into several concatenated bits - the result will be the same. You can aid readability by breaking a long literal into several lines, for instance, with no harm to performance.
If you need the intermediate results of the concatenation for something other than feeding the next iteration of concatenation, StringBuilder isn't going to help you. For instance, if you build up a full name from a first name and a last name, and then add a third piece of information (the nickname, maybe) to the end, you'll only benefit from using StringBuilder if you don't need the (first name + last name) string for other purpose (as we do in the example which creates a Person object).
If you just have a few concatenations to do, and you really want to do them in separate statements, it doesn't really matter which way you go. Which way is more efficient will depend on the number of concatenations the sizes of string involved, and what order they're concatenated in. If you really believe that piece of code to be a performance bottleneck, profile or benchmark it both ways.

C# and beforefieldinit

2008-01-10T06:25:00.000-08:00

The differences between static constructors and type initializers
Some implementations of the singleton pattern rely on the behaviour of static constructors and type initializers, in particular with respect to the time at which they are invoked.

The C# specification (ECMA 334) states in section 17.11:

The static constructor for a class executes at most once in a given application domain. The execution of a static constructor is triggered by the first of the following events to occur within an application domain:

An instance of the class is created.
Any of the static members of the class are referenced.

The CLI specification (ECMA 335) states in section 8.9.5:

A type may have a type-initializer method, or not.
A type may be specified as having a relaxed semantic for its type-initializer method (for convenience below, we call this relaxed semantic BeforeFieldInit)
If marked BeforeFieldInit then the type's initializer method is executed at, or sometime before, first access to any static field defined for that type
If not marked BeforeFieldInit then that type's initializer method is executed at (i.e., is triggered by):
first access to any static or instance field of that type, or
first invocation of any static, instance or virtual method of that type

The C# specification implies that no types with static constructors should be marked with the beforefieldinit flag. Indeed, this is upheld by the compiler, but with a slightly odd effect. I suspect many programmers believe (as I did for a long time) that the following classes were semantically equivalent:

class Test
{
static object o = new object();
}

class Test
{
static object o;

static Test()
{
o = new object();
}
}

The two classes are not, in fact, the same. They both have type initializers - and the two type initializers are the same. However, the first does not have a static constructor, whereas the second does. This means that the first class can be marked as beforefieldinit and have its type initializer invoked at any time before the first reference to a static field in it. The static constructor doesn't even have to do anything. This third class is equivalent to the second:

class Test
{
static object o = new object();

static Test()
{
}
}

I believe this is a source of significant confusion - particularly in terms of singleton implementations.

The curious nature of beforefieldinit - lazy or not?
The beforefieldinit flag has a strange effect, in that it can not only mean that a type initializer is invoked earlier than that of an equivalent type without the flag - it could even be invoked later, or not at all. Consider the following program:

using System;

class Driver
{
public static void Main()
{
Console.WriteLine ("Starting Main");
// Invoke a static method on Test
Test.EchoAndReturn ("Echo!");
Console.WriteLine ("After echo");
// Reference a static member of Test
string y = Test.x;
// Use the value just to avoid compiler cleverness
if (y != null)
Console.WriteLine ("After field access");
}
}

class Test
{
public static string x = EchoAndReturn ("In type initializer");

public static string EchoAndReturn (string s)
{
Console.WriteLine (s);
return s;
}
}

The results of running the above are quite varied. The runtime could decide to run the type initializer on loading the assembly to start with:

In type initializer
Starting Main
Echo!
After echo
After field access

Or perhaps it will run it when the static method is first run...

Starting Main
In type initializer
Echo!
After echo
After field access

Or even wait until the field is first accessed...

Starting Main
Echo!
After echo
In type initializer
After field access

(In theory, the type initializer could even be run after "Echo!" is displayed, but before "After echo" is displayed. I would be very surprised to see any runtime actually show this behaviour, however.) With a static constructor in Test, only the middle of these is possible. So, beforefieldinit can make the invocation of the type initializer even lazier (the last result) or more eager (the first result). I suspect even those developers who know of the existence of beforefieldinit may be surprised by this. The MSDN documentation for TypeAttributes.BeforeFieldInit is particularly poor in this respect. It describes the flag like this:

Specifies that calling static methods of the type does not force the system to initialize the type.

While this is true in the strictest possible sense, it certainly isn't the complete story - it suggests that the flag only makes the initialization lazier, not more eager.

What should be done?
I propose the following changes:

Static field initializers should be treated as if they were part of a static constructor. In other words, any type with a static initializer or an explicit static constructor should not (by default) be marked as beforefieldinit. (Modification to the C# language specification.)
There should be a way of overriding this default behaviour in code. An attribute would be a perfectly reasonable solution to this. (Modification to the C# language specification and addition of an attribute to the standard library.)
The documentation for TypeAttributes.BeforeFieldInit should be clarified significantly. (Modification to MSDN documentation and ECMA 335.)
The above changes are all entirely backwards-compatible, and require no CLI modification.

Further thoughts (after discussion on newsgroups)
The first of the above proposals is definitely the most controversial. (The last isn't controversial at all, as far as I can see.) The reason is performance. Not many classes actually need the behaviour assumed by many C# programmers - most people need never know the difference, really. The JIT compiler, however, cares quite a lot: if a static member is used within a fairly tight loop, for instance, it makes a lot of sense to initialise the type before entering the loop, knowing thereafter that the type has already been initialised. When code is shared between app domains etc, I gather this becomes even more important. Making the performance of existing code decrease by recompilation with a new version of the framework would undoubtedly be unpopular. I'm therefore willing to concede as a less-than-ideal proposal - indeed I've only left it in this page for historical reasons (I dislike the idea of being a revisionist). The second proposal, however, is still important - both to allow classes which do have a static constructor to improve their performance with BeforeFieldInit semantics if appropriate, and to allow classes which currently only need a static constructor to get rid of BeforeFieldInit semantics to achieve this aim in a more self-documenting manner. (A junior developer is more likely to remove a static constructor which appears to be a no-op than to remove an attribute they don't fully understand.)

New features in C# 2.0

2008-01-10T06:24:00.002-08:00

There are dozens (hundreds, probably) of pages listing the new features of C# 2.0. However, I never know where to find a good one quickly, and they don't always tell me what I need to know at the time. I figured if I added my own set of pages, I could update them whenever I wanted to, and point other people at them when answering questions. Without further ado then, here are the new features of C# 2.0:

Various "bits and bobs" described below
Partial types
Aliases
Static classes
Property access modifiers
Nullable types and the null coalescing operator
Delegate changes
Implementing iterators with yield statements
Generics
Partial types
Code generators have existed for a long time. In the past, they usually (depending on the language) either "owned" a whole type/module (creating a whole file which shouldn't or couldn't be edited) or reserved sections of files which shouldn't be edited manually. In some cases, where code was generated by a separate tool from something like a database schema, it could be very hard to make changes to the schema and regenerate the code without losing any additions made by hand.

C# 2.0 introduces the concept of a partial type declaration. This is quite simply a single type which spans multiple files, where each file declares the same type using the partial modifier. The files may refer to members declared within one another without problem (just as forward references within C# is already not a problem). Here's an example (which in itself is a complete program). This allows all the auto-generated code which either mustn't be touched on pain of brokenness or shouldn't be touched because you'll lose all your changes anyway to live in a completely separate file to the code you wish to add. It doesn't help much if you want to tweak the generated code, of course, but that's a less common issue.

Test1.cs:

partial class Test
{
string name;

static void Main()
{
Test t = new Test("C# 2.0");
t.SayHello();
}
}

Test2.cs:

using System;

partial class Test
{
Test(string name)
{
this.name = name;
}

void SayHello()
{
Console.WriteLine ("Hi there. My name is {0}.", name);
}
}

Compile with:

csc Test1.cs Test2.cs

Results:

Hi there. My name is C# 2.0.

A few little things to be aware of:

using directives are only applied to the source file they occur in.
Variable initializers (both instance and static) are executed in textual order, but there's no more guarantee made than that. Given four fields (a1, a2, b1, b2) appearing in files A.cs and B.cs (in the obvious files, with the obvious order), all that is guaranteed is that the initializer for a1 will be executed before the initializer for a2 and b1 before b2. A sequence of a1 b1 b2 a2 is acceptable, although I'd imagine that either a1 a2 b1 b2 or b1 b2 a1 a2 would be more likely.
If a type has a modifier (abstract, public, static etc) applied to it in one place, it has effectively been applied everywhere. In particular, the modifiers on each part of the type definition must not clash - a class cannot be declared as protected in one place and public in another, for example.
Aliases
In previous versions of C#, it was impossible to use two different types which had the same name (including namespace) within the same assembly. (The types themselves would have to be defined in different assemblies anyway, of course, but you might want to use them both from the same assembly.)

C# 2.0 introduces the concept of an "alias". This allows you to effectively name an assembly reference when you compile the code, and use that name to disambiguate between names. As well as disambiguating between identical namespace-qualified names, aliases allow you to disambiguate between names which have been declared within an already used namespace and names which belong to the "root" namespace. This is achieved with the predefined alias of global. Here's an example of aliases at work:

Lib.cs

using System;

namespace Foo.Bar
{
public class Baz
{
public static void SayHiLib()
{
Console.WriteLine ("Hello Lib");
}
}
}

Baz.cs:

using System;

namespace Foo.Bar
{
public class Baz
{
public static void SayHiNested()
{
Console.WriteLine ("Hello Nested");
}
}
}

class Baz
{
public static void SayHiBaz()
{
Console.WriteLine ("Hello Baz");
}
}

Test.cs:

extern alias X;

namespace Foo.Bar
{
class Test
{
static void Main()
{
// Default to using the definition within the same assembly
// and namespace
Baz.SayHiNested();

// Disambiguate to use the definition in the root namespace
global::Baz.SayHiBaz();

// Disambiguate to use the definition in the aliased assembly
X::Foo.Bar.Baz.SayHiLib();
}
}
}

Compile:

csc /target:library Lib.cs
csc /r:X=lib.dll Baz.cs Test.cs

(The results are exactly what you'd expect.)

There are, I suspect, various subtleties to do with aliases. However, I don't know them and I don't want to know them at the moment - because I think aliases should be avoided wherever possible. In a very few cases they'll be absolutely invaluable, but you should really try to avoid the situation where they're needed from cropping up in the first place.

Static classes
Prior to version 2.0, it was impossible to create a class with no instance constructors in C#. If you didn't declare one, the compiler provided a default constructor for you (a public parameterless constructor which called the parameterless constructor of the base type). For classes which were never meant to be instantiated (usually utility classes such as System.Math), this meant you needed to include a private constructor which you didn't call yourself in order to prevent instantiation.

In C# 2.0, there are static classes. These are simply declared using the static modifier. They cannot be derived from or instantiated, and they have no constructors (it is a compile-time error to provide any yourself, and the compiler won't add one for you). Their members must all be static. Here's an example:

using System;

public static class UtilityClass
{
public static void Foo()
{
Console.WriteLine ("Hello");
}

// Uncommenting this creates a compile-time error,
// as static classes can't have instance members
// int x;
}

// Uncommenting this creates a compile-time error,
// as static classes can't be derived from
// public class DerivationAttempt : UtilityClass{}

class Test
{
static void Main()
{
// Uncommenting this creates a compile-time error,
// as static classes cannot be instantiated.
// new UtilityClass();

// You can use static classes in the normal
// way in terms of static members though:
UtilityClass.Foo();
}
}

Property access modifiers
This is a feature which is long, long overdue. Prior to 2.0, it was impossible to declare a property with one access level for the "getter" and a different access level for the "setter". This has meant that people have written separate SetXXX methods if they wanted a public getter but a more limited setter. Fortunately, this glaring omission has been fixed in 2.0. It's very straightforward - you just add the access modifier to the get or set as desired:

public class Test
{
string someProperty;

public string SomeProperty
{
get
{
return someProperty;
}

private set
{
someProperty = value;
}
}
}

The basic rules are that you can't specify an access modifier for both the getter and the setter, and you have to use the "extra" modifier to make access more restrictive than the rest of the property.

Debugging Unicode Problems

2008-01-10T06:24:00.001-08:00

This page describes what to do in a very specific situation. Namely, you've got some character data in one place (typically a database) which has to go through various steps and then ends up being shown to the user (often on a web page). Unfortunately, some characters aren't being displayed correctly. Due to the many steps involved, the problem can occur in various places. This page aims to help you find out what's wrong simply and reliably.

Step 1: Understand the basics of Unicode
If you feel comfortable with Unicode, character encodings etc, feel free to skip this step. Basically, you need to know a little bit about what characters are and what conversions are likely to be applied to them before going much further. See my article on the subject (and the articles it references) for more information.

Step 2: Try to identify the possible conversions involved
If you can work out where things might be going wrong, it's much easier to then isolate which one it is. Also bear in mind not just how you're retrieving the data, but how the data got there in the first place. (Some problems I've seen have been due to an old application writing to and reading from the database in an incorrect way, but the bugs cancelling each other out. No problems occur when it's just this broken application which accesses the database, but things go wrong when anything else does.) Steps involved may well include fetching the data from the database, reading it from a file, sending it across a web connection, or displaying it on the screen.

Step 3: Verify the data at each step
The first lesson here is not to trust anything which tries to log the character data as a sequence of glyphs. Instead, you should log the character data as a sequence of Unicode values (integers). For instance, if I had a string containing the word "hello", I would display it as "0068 0065 006c 006c 006f". (Using hex makes it easier to check values against the Unicode code charts later.) To achieve this, step through each character in the string and display the character however you would display an integer. For instance, here is a method to dump all the characters in a string to the console:

static void DumpString (string value)
{
foreach (char c in value)
{
Console.Write ("{0:x4} ", (int)c);
}
Console.WriteLine();
}

Depending on your exact environment, your method of logging will vary, but using something like the above should give you what you need.

The reason for doing this is that it gets rid of problems with fonts, other encoding issues, etc. If you can't log even plain ASCII hex digits properly, you're in a world of trouble anyway - but you may well not be able to log Unicode in a reliable way, and as you already know you've got some problems on the Unicode front, it's worth being safe.

Now you need to make sure there's a test case to use. Find some (preferrably small) example of where your application is failing, make sure you know exactly what the result should be, and then log the actual result at each of your possible problem points. (Some may be out of your control, but usually if you log as soon as you receive some data and just before you send some data, you'll find the problem.)

Having logged a problematic string, you should verify whether or not it's what it should be. This is where the Unicode code charts page comes in. You can either pick which block you believe the correct character is in, or you can search for your character alphabetically. Check that each character in the string has its proper Unicode value. As soon as you find a point in your application flow where the character data is corrupted, you should investigate that area of the code, find out why it's being corrupted and fix it. When you've got it right throughout the application flow, the application should be working properly.

Plug-ins and cast exceptions

2008-01-10T06:23:00.000-08:00

A common error people post about in the .NET newsgroups is an InvalidCastException being thrown when loading types dynamically, typically for a plug-in mechanism. This article explains the most common reason, and how to avoid it.

The situation
Generally, there are three main types involved in this kind of set-up: the "driver" class (which loads the plug-in assembly), the interface (which plug-ins must implement), and the plug-in class itself. In order to simplify the example, I'm ignoring the normal code which looks for appropriate types within an assembly, and indeed appropriate assemblies within a directory etc. Here is the source code for the three types involved:

Driver.cs:

using System;
using System.Reflection;

public class Driver
{
static void Main()
{
Assembly assembly = Assembly.LoadFrom ("myplugin.dll");
Type t = assembly.GetType ("SamplePlugin");
IPlugin plugin = (IPlugin) Activator.CreateInstance(t);
plugin.SayHello();
}
}

IPlugin.cs:

public interface IPlugin
{
void SayHello();
}

SamplePlugin.cs:

using System;

public class SamplePlugin : IPlugin
{
public void SayHello()
{
Console.WriteLine ("Hello from the sample plug-in.");
}
}

How not to compile the code
With those files all in the same directory, you can compile them with:

csc Driver.cs IPlugin.cs
csc /target:library /out:myplugin.dll IPlugin.cs SamplePlugin.cs

That produces two assemblies: Driver.exe and myplugin.dll. This corresponds to having two different projects in Visual Studio .NET, and both projects containing IPlugin.cs. Unfortunately, when you run it, you get:

Unhandled Exception: System.InvalidCastException: Specified cast is not valid.
at Driver.Main()

What's going wrong?
The cast in the driver code is saying, "Check that the reference actually points to an instance of IPlugin (or it's null), and throw an exception if it doesn't." However, the IPlugin type that it is trying to check against is the one that Driver knows about - the one within the Driver.exe assembly. Unfortunately, the actual type of the object in question is SamplePlugin which implements the IPlugin type which is in myplugin.dll. Even though the names are the same and the contents are the same, the runtime treats them as completely different types. A type is uniquely identified by its fully qualified name and the assembly it comes from. The runtime keeps track of which assembly files it has loaded, but if there were two identical assemblies in two separate files, those would be loaded as two different assemblies - so the types within them would be completely separate.

How can we fix it?
Clearly we need to make sure that the type used for casting is the same one which the object itself is aware of. Basically, we want to make sure that there's only one IPlugin type in memory. There are two simple ways of doing this, although one of them won't work in Visual Studio .NET:

Solution 1: Keep the interface with the driver assembly, and refer to it
Change the compilation process to:

csc Driver.cs IPlugin.cs
csc /target:library /out:myplugin.dll /r:Driver.exe SamplePlugin.cs

Here we have the same assemblies as before, but this time instead of IPlugin being in the plug-in assembly as well as the driver, it's only in the driver, and we tell the compiler to reference the driver assembly when compiling the plug-in. Running it, we get the desired output:

Hello from the sample plug-in.

Unfortunately, this won't work from Visual Studio .NET 2003 - at least, not with the "Add Reference" dialog. Even though the runtime lets an assembly reference an executable assembly, VS.NET 2003 doesn't - if you try to add a reference to Driver.exe, you get an error message. Fortunately, this restriction has been lifted in VS 2005, making this solution or more appealing one. A "newbie trying to help" pointed out to me that you can, however, edit the project file in VS.NET 2003 to add the reference manually. Here's an example:

Name = "IPlugin"
AssemblyName = "PlugInTest"
HintPath = "..\PlugInTest\bin\debug\plugintest.exe"
/>

Solution 2: Separate the interface into its own assembly
Change the compilation process to:

csc /target:library /out:interface.dll IPlugin.cs
csc /r:interface.dll Driver.cs
csc /target:library /out:myplugin.dll /r:interface.dll SamplePlugin.cs

This is equivalent to there now being three projects in Visual Studio .NET - one for each type. The driver and sample plug-in projects need to reference the project providing the plug-in interface. Three assemblies are now generated, interface.dll, Driver.exe and myplugin.dll. Again, the interface only exists in a single assembly in memory, so both the driver casting code and the plug-in class itself will use the same type. The output is as with solution 1 - no exceptions are thrown.

Conclusion
When writing plug-in architectures, make absolutely sure that each type is only present in a single assembly. Apart from some extreme cases (for instance where the same file is loaded multiple times as different assemblies) this will be enough to get rid of any InvalidCastException you may be experiencing. If you still have problems, please either mail me at skeet@pobox.com, or post in the newsgroup. Given that these situations are frequently complicated, it helps to have some short and complete example code to demonstrate the problem.

Binary floating point and .NET

2008-01-10T06:22:00.000-08:00

Lots of people are at first surprised when some of their arithmetic comes out "wrong" in .NET. This isn't something specific to .NET in particular - most languages/platforms use something called "floating point" arithmetic for representing non-integer numbers. This is fine in itself, but you need to be a bit aware of what's going on under the covers, otherwise you'll be surprised at some of the results.

It's worth noting that I am not an expert on this matter. Since writing this article, I've found another one - this time written by someone who really is an expert, Jeffrey Sax. I strongly recommend that you read his article on floating point concepts too.

What is floating point?
Computers always need some way of representing data, and ultimately those representations will always boil down to binary (0s and 1s). Integers are easy to represent (with appropriate conventions for negative numbers, and with well-specified ranges to know how big the representation is to start with) but non-integers are a bit more tricky. Whatever you come up with, there'll be a problem with it. For instance, take out own normal way of writing numbers in decimal: that can't (in itself) express a third. You end up with a recurring 3. Whatever base you come up with, you'll have the same problem with some numbers - and in particular, "irrational" numbers (numbers which can't be represented as fractions) like the mathematical constants pi and e are always going to give trouble.

You could store all the rational numbers exactly as two integers, with the number being the first number divided by the second - but the integers can grow quite large quite quickly even for "simple" operations, and things like square roots will tend to produce irrational numbers. There are various other schemes which also pose problems, but the one most systems use in one form or other is floating point. The idea of this is that basically you have one integer (the mantissa) which gives some scaled representation of the number, and another (the exponent) which says what the scale is, in terms of "where does the dot go". For instance, 34.5 could be represented in "decimal floating point" as mantissa 3.45 with an exponent of 1, whereas 3450 would have the same mantissa but an exponent of 3 (as 34.5 is 3.45x101, and 3450 is 3.45x103). Now, that example is in decimal just for simplicity, but the most common formats of floating point are for binary. For instance, the binary mantissa 1.1 with an exponent of -1 would mean decimal 0.75 (binary 1.1==decimal 1.5, and the exponent of -1 means "divide by 2" in the same way that a decimal exponent of -1 means "divide by 10").

It's very important to understand that in the same way that you can't represent a third exactly in a (finite) decimal expansion, there are lots of numbers which look simple in decimal, but which have long or infinite expansions in a binary expansion. This means that (for instance) a binary floating point variable can't have the exact value of decimal 0.1. Instead, if you have some code like

double x = 0.1d;
the variable will actually store the closest available double to that value. Once you can get your head round that, it becomes obvious why some calculations seem to be "wrong". If you were asked to add a third to a third, but could only represent the thirds using 3 decimal places, you'd get the "wrong" answer: the closest you could get to a third is 0.333, and adding two of those together gives 0.666, rather than 0.667 (which is closer to the exact value of two thirds). An example in binary floating point is that 3.65d+0.05d != 3.7d (although it may be displayed as 3.7 in some situations).

What floating point types are available in .NET?
The C# standard only lists double and float as floating points available (those being the C# shorthand for System.Double and System.Single), but I believe that strictly speaking the decimal type (shorthand for System.Decimal) is also a floating point type - it's just it's decimal floating point, and the ranges of exponents are interesting. The decimal type is described in another article, so this doesn't go into it any further - we're concentrating on the more commonly used double and float. Both of these are binary floating point types, conforming to IEEE 754 (a standard defining various floating point types). float is a 32 bit type (1 bit of sign, 23 bits of mantissa, and 8 bits of exponent), and double is a 64 bit type (1 bit of sign, 52 bits of mantissa and 11 bits of exponent).

Isn't it bad that results aren't what I'd expect?
Well, that depends on the situation. If you're writing financial applications, you probably have very rigidly defined ways of treating errors, and the amounts are also intuitively represented as decimal - in which case the decimal type is more likely to be appropriate than float or double. If, however, you're writing a scientific app, the link with the decimal representation is likely to be weaker, and you're also likely to be dealing with less precise amounts to start with (a dollar is exactly a dollar, but if you've measured a length to be a metre, that's likely to have some sort of inaccuracy in it to start with).

Comparing floating point numbers
One consequence of all of this is that you should very, very rarely be comparing binary floating point numbers for equality directly. It's usually fine to compare in terms of greater-than or less-than, but when you're interested in equality you should always consider whether what you actually want is near equality: is one number almost the same as another. One simple way of doing this is to subtract one from the other, use Math.Abs to find the absolute value of the difference, and then check whether this is lower than a certain tolerance level.

There are some cases which are particularly pathological though, and these are due to JIT optimisations. Look at the following code:

using System;

class Test
{
static float f;

static void Main(string[] args)
{
f = Sum (0.1f, 0.2f);
float g = Sum (0.1f, 0.2f);
Console.WriteLine (f==g);
// g = g+1;
}

static float Sum (float f1, float f2)
{
return f1+f2;
}
}

It should always print True, right? Wrong, unfortunately. When running under debug, where the JIT can't make as many optimisations as normal, it will print True. When running normally, the JIT can store the result of the sum more accurately than a float can really represent - it can use the default x86 80-bit representation, for instance, for the sum itself, the return value, and the local variable. See the ECMA CLI spec, partition 1, section 12.1.3 for more details. Uncommenting the commented out line in the above, makes the JIT behave a bit more conservatively - the result is then True either way - although that may only be true under the current implementation, and shouldn't be relied on. (Casting g to float in the comparison line has the same effect, even though it looks like a no-op.) This is another reason to avoid equality comparisons even if you're really sure that the results should be the same.

How does .NET format floating point numbers?
There's no built-in way to see the exact decimal value of a floating point number in .NET, although you can do it with a bit of work. (See the bottom of this article for some code to do this.) By default, .NET formats a double to 15 decimal places, and a float to 7. (In some cases it will use scientific notation; see the MSDN page on standard numeric format strings for more information.) If you use the round-trip format specifier ("r"), it formats the number to the shortest form which, when parsed (to the same type), will get back to the original number. If you are storing floating point numbers as strings and the exact value is important to you, you should definitely use the round-trip specifier, as otherwise you are very likely to lose data.

What exactly does a floating point number look like in memory?
As it says above, a floating point number basically has a sign, an exponent and a mantissa. All of these are integers, and the combination of the three of them specifies exactly what number is being represented. There are various classes of floating point number: normalised, subnormal, infinity and not a number (NaN). Most numbers are normalised, which means that the first bit of the binary mantissa is assumed to be 1, which means you don't actually need to store it. For instance, the binary number 1.01101 could be expressed as just .01101 - the leading 1 is assumed, as if it were 0 a different exponent would be used. That technique only works when the number is in the range where you can choose the exponent suitably. Numbers which don't lie in that range (very, very small numbers) are called subnormal, and no leading bit is assumed. "Not a number" (NaN) values are for things like the result of dividing 0 by 0, etc. There are various different classes of NaN, and there's some odd behaviour there as well. Subnormal numbers are also sometimes called denormalised numbers.

The actual representation of the sign, exponent and mantissa at the bit level is for each of them to be an unsigned integer, with the stored value being just the concatenation of the sign, then the exponent, then the mantissa. The "real" exponent is biased - for instance, in the case of a double, the exponent is biased by 1023, so a stored exponent value of 1026 really means 3 when you come to work out the actual value. The table below shows what each combination of sign, exponent and mantissa means, using double as an example. The same principles apply for float, just with slightly different values (such as the bias). Note that the exponent value given here is the stored exponent, before the bias is applied. (That's why the bias is shown in the "value" column.)

Sign (s, 1 bit) Stored exponent (e, 11 bits) Mantissa (m, 52 bits) Type of number Value
Any Non-zero Any Normal (-1)s x 1.m (binary) x 2e-1023
0 0 0 Zero +0
1 0 0 Zero +0
0 2047 0 Infinity Positive infinity
1 2047 0 Infinity Negative infinity
0 2047 Non-zero NaN n/a

Worked example
Consider the following 64-bit binary number:
0100000001000111001101101101001001001000010101110011000100100011

As a double, this is split into:

Sign: 0
Exponent: 10000000100 binary = 1028 decimal
Mantissa: 0111001101101101001001001000010101110011000100100011
This is therefore a normal number of value
(-1)0 x 10111001101101101001001001000010101110011000100100011 (binary) x 21028-1023
which is more simply represented as
1.0111001101101101001001001000010101110011000100100011 (binary) x 25
or
101110.01101101101001001001000010101110011000100100011

In decimal, this is 46.42829231507700882275457843206822872161865234375, but .NET will display it by default as 46.428292315077 or with the "round-trip" format specifier as 46.428292315077009.

Sample code
DoubleConverter.cs: this is a fairly simple class which allows you to convert a double to its exact decimal representation as a string. Note that although finite decimals don't always have finite binary expansions, all finite binaries have a finite decimal expansion (because 2 is a factor of 10, essentially). The class is extremely simple to use - just call DoubleConverter.ToExactString(value) and the exact string representation for value is returned. There's also a simple online version - enter the number as you'd write it in code, and the exact value which is actually used will be displayed.

NaNs
NaNs are odd beasts. There are two types of NaNs - signalling and quiet, or SNaN and QNaN for short. In terms of the bit pattern, a quiet NaN has the top bit of the mantissa set, whereas a signalling NaN has it cleared. Quiet NaNs are used to signify that the result of a mathematical operation is undefined, whereas signalling NaNs are used to signify exceptions (where the operation was invalid, rather than just having an indeterminate outcome).

The strangest thing most people are likely to see about NaNs is that they're not equal to themselves. For instance, Double.NaN==Double.NaN evaluates to false. Instead, you need to use Double.IsNaN to check whether a value is not a number. Fortunately, most people are unlikely to encounter NaNs at all except in articles like this.

Conclusion
Binary floating point arithmetic is fine so long as you know what's going on and don't expect values to be exactly the decimal ones you put in your program, and don't expect calculations involving binary floating point numbers to necessarily yield precise results. Even if two numbers are both exactly represented in the type you're using, the result of an operation involving those two numbers won't necessarily be exactly represented. This is most easily seen with division (eg 1/10 isn't exactly representable despite both 1 and 10 being exactly representable) but it can happen with any operation - even seemingly innocent ones such as addition and subtraction.

If you particularly want precise decimal numbers, consider using the decimal type instead - but expect to pay a performance penalty for doing so. (One very quickly devised test came out with multiplication of doubles being about 40 times faster than multiplication of decimals; don't pay particular heed to this exact figure, but take it as an indication that binary floating point is generally faster on current hardware than decimal floating point.)

Decimal floating point in .NET

2008-01-10T06:21:00.002-08:00

In my article on binary floating point types, I mentioned the System.Decimal (or just decimal in C#) type briefly. This article gives more details about the type, including its representation and some differences between it and the more common binary floating point types. From here on, I shall just refer to it as the decimal type rather than System.Decimal, and likewise where float and double are mentioned, I mean the .NET types System.Single and System.Double respectively. To make the article easier on the eyes, I'll leave the names in normal type from here on, too.

What is the decimal type?
The decimal type is just another form of floating point number - but unlike float and double, the base used is 10. If you haven't read the article linked above, now would be a good time to read it - I won't go into the basics of floating point numbers in this article.

The decimal type has the same components as any other floating point number: a mantissa, an exponent and a sign. As usual, the sign is just a single bit, but there are 96 bits of mantissa and 5 bits of exponent. However, not all exponent combinations are valid. Only values 0-28 work, and they are effectively all negative: the numeric value is sign * mantissa / 10exponent. This means the maximum and minimum values of the type are +/- (296-1), and the smallest non-zero number in terms of absolute magnitude is 10-28.

The reason for the exponent being limited is that the mantissa is able to store 28 or 29 decimal digits (depending on its exact value). Effectively, it's as if you have 28 digits which you can set to any value you want, and you can put the decimal point anywhere from the left of the first digit to the right of the last digit. (There are some numbers where you can have a 29th digit to the left of the rest, but you can't have all combinations with 29 digits, hence the restriction.)

How is a decimal stored?
A decimal is stored in 128 bits, even though only 102 are strictly necessary. It is convenient to consider the decimal as three 32-bit integers representing the mantissa, and then one integer representing the sign and exponent. The top bit of the last integer is the sign bit (in the normal way, with the bit being set (1) for negative numbers) and bits 16-23 (the low bits of the high 16-bit word) contain the exponent. The other bits must all be clear (0). This representation is the one given by decimal.GetBits(decimal) which returns an array of 4 ints.

Formatting decimals
Unlike floats and doubles, when .NET is asked to format a decimal into a string representation, its default behaviour is to give the exact value. This means there is no need for a decimal equivalent of the DoubleConverter code of the binary floating point article. You can, of course, ask it to restrict the value to a specific precision.

Keeping zeroes
Between .NET 1.0 and 1.1, the decimal type underwent a subtle change. Consider the following simple program:

using System;

public class Test
{
static void Main()
{
decimal d = 1.00m;
Console.WriteLine (d);
}
}

When I first ran the above (or something similar) I expected it to output just 1 (which is what it would have been on .NET 1.0) - but in fact, the output was 1.00. The decimal type doesn't normalize itself - it remembers how many decimal digits it has (by maintaining the exponent where possible) and on formatting, zero may be counted as a significant decimal digit. I don't know the exact nature of what exponent is chosen (where there is a choice) when two different decimals are multiplied, divided, added etc, but you may find it interesting to play around with programs such as the following:

using System;

public class Test
{
static void Main()
{
decimal d = 0.00000000000010000m;
while (d != 0m)
{
Console.WriteLine (d);
d = d/5m;
}
}
}

Which produces a result of:

0.00000000000010000
0.00000000000002000
0.00000000000000400
0.00000000000000080
0.00000000000000016
0.000000000000000032
0.0000000000000000064
0.00000000000000000128
0.000000000000000000256
0.0000000000000000000512
0.00000000000000000001024
0.000000000000000000002048
0.0000000000000000000004096
0.00000000000000000000008192
0.000000000000000000000016384
0.0000000000000000000000032768
0.0000000000000000000000006554
0.0000000000000000000000001311
0.0000000000000000000000000262
0.0000000000000000000000000052
0.000000000000000000000000001
0.0000000000000000000000000002

Everything's a number
The decimal type has no concept of infinity or NaN (not-a-number) values, and despite the above examples of the same actual number being potentially representable in different forms (eg 1, 1.0, 1.00) the normal == operator copes with these and reports 1.0==1.00 etc.

Accuracy
The decimal type has a larger precision than any of the built-in binary floating point types in .NET, although it has a smaller range of potential exponents. Also, many operations which yield surprising results in binary floating point due to inexact representations of the original operands go away in decimal floating point, precisely because many operands are specifically represented in source code as decimals. However, that doesn't mean that all operations suddenly become accurate: a third still isn't exactly representable, for instance. The potential problems are just the same as they are with binary floating point. However, most of the time the decimal type is chosen for quantities like money, where operations will be simple and keep things accurate. (For instance, adding a tax which is specified as a percentage will keep the numbers accurate, assuming they're in a sensible range to start with.) Just be aware of which operations are likely to cause inaccuracy, and which aren't. As a very broad rule of thumb, if you end up seeing a very long string representation (ie most of the 28/29 digits are non-zero) then chances are you've got some inaccuracy along the way: most of the uses of the decimal type won't end up using very many significant figures when the numbers are exact. If you find yourself using inaccurate numbers, you should make sure that you expected it, and consider why you're using the decimal type in the first place. (In some situations it'll make sense despite the performance hit; in many it won't.)

Reading binary data in C#

2008-01-10T06:21:00.001-08:00

In the C# newsgroup, I've seen quite a lot of code for reading in data from a file like this:

// Bad code! Do not use!
FileStream fs = File.OpenRead(filename);
byte[] data = new byte[fs.Length];
fs.Read (data, 0, data.Length);

This code is far from guaranteed to work. In particular, the FileStream could be reading just the first 10 bytes of the file into the buffer. The Read method is only guaranteed to block until some data is available (or the end of the stream is reached), not until all of the data is available. That's where the return value (which is ignored in the above code) is vital. You need to cope with the case where you can't read all of the data in one go, and loop round until you've read what you want. Here's a method which you can use if you want to read from a stream into the whole of an array, not stopping until it's finished:

///

/// Reads data into a complete array, throwing an EndOfStreamException
/// if the stream runs out of data first, or if an IOException
/// naturally occurs.
///

/// The stream to read data from
/// The array to read bytes into. The array
/// will be completely filled from the stream, so an appropriate
/// size must be given.
public static void ReadWholeArray (Stream stream, byte[] data)
{
int offset=0;
int remaining = data.Length;
while (remaining > 0)
{
int read = stream.Read(data, offset, remaining);
if (read <= 0)
throw new EndOfStreamException
(String.Format("End of stream reached with {0} bytes left to read", remaining));
remaining -= read;
offset += read;
}
}

Sometimes, you don't know the length of the stream in advance (for instance a network stream) and just want to read the whole lot into a buffer. Here's a method to do just that:

///

/// Reads data from a stream until the end is reached. The
/// data is returned as a byte array. An IOException is
/// thrown if any of the underlying IO calls fail.
///

/// The stream to read data from
public static byte[] ReadFully (Stream stream)
{
byte[] buffer = new byte[32768];
using (MemoryStream ms = new MemoryStream())
{
while (true)
{
int read = stream.Read (buffer, 0, buffer.Length);
if (read <= 0)
return ms.ToArray();
ms.Write (buffer, 0, read);
}
}
}

While the above is simple, it's not terribly efficient, as it ends up copying the data at the very end, and probably several times between. Here's some code which works well if you know the expected length of data to start with. (While you could use Stream.Length, it isn't supported for all streams.)

///

/// Reads data from a stream until the end is reached. The
/// data is returned as a byte array. An IOException is
/// thrown if any of the underlying IO calls fail.
///

/// The stream to read data from
/// The initial buffer length
public static byte[] ReadFully (Stream stream, int initialLength)
{
// If we've been passed an unhelpful initial length, just
// use 32K.
if (initialLength < 1)
{
initialLength = 32768;
}

byte[] buffer = new byte[initialLength];
int read=0;

int chunk;
while ( (chunk = stream.Read(buffer, read, buffer.Length-read)) > 0)
{
read += chunk;

// If we've reached the end of our buffer, check to see if there's
// any more information
if (read == buffer.Length)
{
int nextByte = stream.ReadByte();

// End of stream? If so, we're done
if (nextByte==-1)
{
return buffer;
}

// Nope. Resize the buffer, put in the byte we've just
// read, and continue
byte[] newBuffer = new byte[buffer.Length*2];
Array.Copy(buffer, newBuffer, buffer.Length);
newBuffer[read]=(byte)nextByte;
buffer = newBuffer;
read++;
}
}
// Buffer is now too big. Shrink it.
byte[] ret = new byte[read];
Array.Copy(buffer, ret, read);
return ret;
}

Using code such as the above, whether synchronously or asynchronously, you shouldn't come across the kinds of error that can otherwise occur, such as data which appears to be corrupted or truncated.

Multi-threading in .NET: Introduction and suggestions

2008-01-10T06:18:00.000-08:00

One of the greatest understatements I've heard in a newsgroup was made by Patricia Shanahan, in a Java newsgroup in 2001: "Multi-threaded programming needs a little care." Multi-threading is probably one of the worst understood aspects of programming, and these days almost all application programmers need to understand it to some extent. This article acts as an introduction to multi-threading and gives some hints and tips for how to do it safely. Warning: I'm not an expert on the subject, and when the real experts start discussing it in detail, my head starts to spin somewhat. However, I've tried to pay attention to those who know what they're doing, and hopefully the contents of this article form at least part of a multi-threading "best practice".

This article uses the C# type shorthands throughout - int for Int32 etc. I hope this makes it easier for C# developers to read, and won't impede any other developers too much. It also only talks about the C# ways of declaring variables to be volatile and locking monitors. Developers using other languages can find the equivalents in their own preferred environment, I'm sure.

Introduction: What is multi-threading?
The fact that you're reading this article in the first place means you probably have at least some idea of what multi-threading is about: it's basically trying to do more than one thing at a time within a process.

So, what is a thread? A thread (or "thread of execution") is a sort of context in which code is running. Any one thread follows program flow for wherever it is in the code, in the obvious way. Before multi-threading, effectively there was always one thread running for each process in an operating system (and in many systems, there was only one process running anyway). If you think of processes running in parallel in an operating system (e.g. a browser downloading a file and a word processor allowing you to type, both "at the same time"), then apply the same kind of thinking within a single process, that's a reasonable way to visualise threading.

Multi-threading can occur in a "real" sense, in that a multi-processor box may have more than one processor executing instructions for a particular process at a time, or it may be effectively "simulated" by multiple threads executing in sequence: first some code for thread 1 is executed, then some code for thread 2, then back to thread 1 etc. In this situation, if both thread 1 and thread 2 are "compute bound" (all they're doing is computation, without waiting for any input from the network, or file system, or user etc) then that won't actually speed things up at all - in fact, it'll slow things down as the operating system has to switch between threads, and the memory cache probably won't be as effective. However, much of today's computing involves waiting for something to happen, and during that time the processor can be doing something else. Intel's "Hyper-Threading" technology which is on some of its more recent chips (bearing in mind that this article was written in early 2004!) is a sort of hybrid between this "real" and "simulated" threading - for more information, see Intel's web page on the subject.

How does multi-threading work in .NET?
.NET has been designed from the start to support multi-threaded operation. There are two main ways of multi-threading which .NET encourages: starting your own threads with ThreadStart delegates, and using the ThreadPool class either directly (using ThreadPool.QueueUserWorkItem) or indirectly using asynchronous methods (such as Stream.BeginRead, or calling BeginInvoke on any delegate).

In general, you should create a new thread "manually" for long-running tasks, and use the thread pool only for brief jobs. The thread pool can only run so many jobs at once, and some framework classes use it internally, so you don't want to block it with a lot of tasks which need to block for other things. The examples in this article mostly use manual thread creation. On the other hand, for short-running tasks, particularly those created often, the thread pool is an excellent choice.

Multi-threaded "Hello, world"
Here is virtually the simplest threading example which actually shows something happening:

using System;
using System.Threading;

public class Test
{
static void Main()
{
ThreadStart job = new ThreadStart(ThreadJob);
Thread thread = new Thread(job);
thread.Start();

for (int i=0; i < 5; i++)
{
Console.WriteLine ("Main thread: {0}", i);
Thread.Sleep(1000);
}
}

static void ThreadJob()
{
for (int i=0; i < 10; i++)
{
Console.WriteLine ("Other thread: {0}", i);
Thread.Sleep(500);
}
}
}

The code creates a new thread which runs the ThreadJob method, and starts it. That thread counts from 0 to 9 fairly fast (about twice a second) while the main thread counts from 0 to 4 fairly slowly (about once a second). The way they count at different speeds is by each of them including a call to Thread.Sleep, which just makes the current thread sleep (do nothing) for the specified period of time. Between each count in the main thread we sleep for 1000ms, and between each count in the other thread we sleep for 500ms. Here are the results from one test run on my machine:

Main thread: 0
Other thread: 0
Other thread: 1
Main thread: 1
Other thread: 2
Other thread: 3
Main thread: 2
Other thread: 4
Other thread: 5
Main thread: 3
Other thread: 6
Other thread: 7
Main thread: 4
Other thread: 8
Other thread: 9

One important thing to note here is that although the above is very regular, that's by chance. There's nothing to stop the first "Other thread" line coming first, or the pattern being slightly off - Thread.Sleep is always going to be somewhat approximate, and there's no guarantee that the sleeping thread will immediately start running as soon as the sleep finishes. (It will become able to run, but another thread may be currently running, and on a single processor machine that means the thread which has just "woken up" will have to wait until the thread scheduler decides to give it some processor time before it next does anything.)

As with all delegates, there's nothing to restrict you to static methods, or methods within the class that the delegate is used from. You need to have access to the method, of course, and if you want to specify an instance method, you have to use a particular instance. Here's another version of the program above, using an instance method in a different class. If the Count method had been static, the value of the job variable would have been new ThreadStart(Counter.Count). Most examples given in this article use methods within the same class, but that's just for brevity and simplicity.

using System;
using System.Threading;

public class Test
{
static void Main()
{
Counter foo = new Counter();
ThreadStart job = new ThreadStart(foo.Count);
Thread thread = new Thread(job);
thread.Start();

for (int i=0; i < 5; i++)
{
Console.WriteLine ("Main thread: {0}", i);
Thread.Sleep(1000);
}
}
}

public class Counter
{
public void Count()
{
for (int i=0; i < 10; i++)
{
Console.WriteLine ("Other thread: {0}", i);
Thread.Sleep(500);
}
}
}

Strings in .NET and C#

2008-01-10T06:17:00.000-08:00

The System.String type (shorthand string in C#) is one of the most important types in .NET, and unfortunately it's much misunderstood. This article attempts to deal with some of the basics of the type.

What is a string?
A string is basically a sequence of characters. Each character is a Unicode character in the range U+0000 to U+FFFF (more on that later). The string type (I'll use the C# shorthand rather than putting System.String each time) has the following characteristics:

It is a reference type
It's a common misconception that string is a value type. That's because its immutability (see next point) makes it act sort of like a value type. It actually acts like a normal reference type. See my articles on parameter passing and memory for more details of the differences between value types and reference types.
It's immutable
You can never actually change the contents of a string, at least with safe code which doesn't use reflection. Because of this, you often end up changing the value of a string variable. For instance, the code s = s.Replace ("foo", "bar"); doesn't change the contents of the string that s originally referred to - it just sets the value of s to a new string, which is a copy of the old string but with "foo" replaced by "bar".
It can contain nulls
C programmers are used to strings being sequences of characters ending in '\0', the nul or null character. (I'll use "null" because that's what the Unicode code chart calls it in the detail; don't get it confused with the null keyword in C# - char is a value type, so can't be a null reference!) In .NET, strings can contain null characters with no problems at all as far as the string methods themselves are concerned. However, other classes (for instance many of the Windows Forms ones) may well think that the string finishes at the first null character - if your string ever appears to be truncated oddly, that could be the problem.
It overloads the == operator
When the == operator is used to compare two strings, the Equals method is called, which checks for the equality of the contents of the strings rather than the references themselves. For instance, "hello".Substring(0, 4)=="hell" is true, even though the references on the two sides of the operator are different (they refer to two different string objects, which both contain the same character sequence). Note that operator overloading only works here if both sides of the operator are string expressions at compile time - operators aren't applied polymorphically. If either side of the operator is of type object as far as the compiler is concerned, the normal == operator will be applied, and simple reference equality will be tested.
Interning
.NET has the concept of an "intern pool". It's basically just a set of strings, but it makes sure that every time you reference the same string literal, you get a reference to the same string. This is probably language-dependent, but it's certainly true in C# and VB.NET, and I'd be very surprised to see a language it didn't hold for, as IL makes it very easy to do (probably easier than failing to intern literals). As well as literals being automatically interned, you can intern strings manually with the Intern method, and check whether or not there is already an interned string with the same character sequence in the pool using the IsInterned method. This somewhat unintuitively returns a string rather than a boolean - if an equal string is in the pool, a reference to that string is returned. Otherwise, null is returned. Likewise, the Intern method returns a reference to an interned string - either the string you passed in if was already in the pool, or a newly created interned string, or an equal string which was already in the pool.

Literals
Literals are how you hard-code strings into C# programs. There are two types of string literals in C# - regular string literals and verbatim string literals. Regular string literals are similar to those in many other languages such as Java and C - they start and end with ", and various characters (in particular, " itself, \, and carriage return (CR) and line feed (LF)) need to be "escaped" to be represented in the string. Verbatim string literals allow pretty much anything within them, and end at the first " which isn't doubled. Even carriage returns and line feeds can appear in the literal! To obtain a " within the string itself, you need to write "". Verbatim string literals are distinguished by having an @ before the opening quote. Here are some examples of the two types of literal, and what they amount to:

Regular literal Verbatim literal Resulting string
"Hello" @"Hello" Hello
"Backslash: \\" @"Backslash: \" Backslash: \
"Quote: \"" @"Quote: """ Quote: "
"CRLF:\r\nPost CRLF" @"CRLF:
Post CRLF" CRLF:
Post CRLF

For other escape sequences, please see the relevant FAQ entry. Note that the difference is only for the compiler's sake. Once the string is in the compiled code, there's no such thing as a verbatim string literal vs a regular string literal.

Strings and the debugger
Numerous people run into problems when inspecting strings in the debugger, both with VS.NET 2002 and VS.NET 2003. Ironically, the problems are often generated by the debugger trying to be helpful, and either displaying the string as a regular string literal with backslash-escaped characters in, or displaying it as a verbatim string literal complete with leading @. This leads to many questions asking how the @ can be removed, despite the fact that it's not really there in the first place - it's only how the debugger's showing it. Also, some versions of VS.NET will stop displaying the contents of the string at the first null character, and evaluate its Length property incorrectly, calculating the value itself instead of asking the managed code. Again, it then considers the string to finish at the first null character.

Given the confusion this has caused, I believe it's best to examine strings in a different way when debugging, at least if you think something odd is going on. I suggest using a method like the one below, which will print the contents of a string to the console in a safe way. Depending on what kind of application you're developing, you may want to write this information to a log file, to the debug or trace listeners, or pop it up in a message box.

static readonly string[] LowNames =
{
"NUL", "SOH", "STX", "ETX", "EOT", "ENQ", "ACK", "BEL",
"BS", "HT", "LF", "VT", "FF", "CR", "SO", "SI",
"DLE", "DC1", "DC2", "DC3", "DC4", "NAK", "SYN", "ETB",
"CAN", "EM", "SUB", "ESC", "FS", "GS", "RS", "US"
};
public static void DisplayString (string text)
{
Console.WriteLine ("String length: {0}", text.Length);
foreach (char c in text)
{
if (c < 32)
{
Console.WriteLine ("<{0}> U+{1:x4}", LowNames[c], (int)c);
}
else if (c > 127)
{
Console.WriteLine ("(Possibly non-printable) U+{0:x4}", (int)c);
}
else
{
Console.WriteLine ("{0} U+{1:x4}", c, (int)c);
}
}
}

Memory usage
In the current implementation at least, strings take up 20+(n/2)*4 bytes (rounding the value of n/2 down), where n is the number of characters in the string. The string type is unusual in that the size of the object itself varies. The only other classes which do this (as far as I know) are arrays. Essentially, a string is a character array in memory, plus the length of the array and the length of the string (in characters). The length of the array isn't always the same as the length in characters, as strings can be "over-allocated" within mscorlib.dll, to make building them up easier. (StringBuilder does this, for instance.) While strings are immutable to the outside world, code within mscorlib can change the contents, so StringBuilder creates a string with a larger internal character array than the current contents requires, then appends to that string until the character array is no longer big enough to cope, at which point it creates a new string with a larger array. The string length member also contains a flag in its top bit to say whether or not the string contains any non-ASCII characters. This allows for extra optimisation in some cases.

Although strings aren't null-terminated as far as the API is concerned, the character array is null-terminated, as this means it can be passed directly to unmanaged functions without any copying being involved, assuming the inter-op specifies that the string should be marshalled as Unicode.

Encoding
(If you don't know about character encodings and Unicode, please read my article on the subject first.)

As stated at the start of the article, strings are always in Unicode encoding. The idea of "a Big-5 string" or "a string in UTF-8 encoding" is a mistake (as far as .NET is concerned) and usually indicates a lack of understanding of either encodings or the way .NET handles strings. It's very important to understand this - treating a string as if it represented some valid text in a non-Unicode encoding is almost always a mistake.

Now, the Unicode coded character set (one of the flaws of Unicode is that the one term is used for various things, including a coded character set and a character encoding scheme) contains more than 65536 characters. This means that a single char (System.Char) cannot cover every character. This leads to the use of surrogates where characters above U+FFFF are represented in strings as two characters. Essentially, string uses the UTF-16 character encoding form. Most developers may well not need to know much about this, but it's worth at least being aware of it.

Culture and internationalization oddities
Some of the oddities of Unicode lead to oddities in string and character handling. Many of the string methods are culture-sensitive - in other words, what they do depends on the culture of the current thread. For example, what would you expect "i".toUpper() to return? Most people would say "I", but in Turkish the correct answer is "İ" (Unicode U+0130, "Latin capital I with dot above"). To perform a culture-insensitive case change, you can use CultureInfo.InvariantCulture, and pass that to the overload of String.ToUpper which takes a CultureInfo.

There are further oddities when it comes to comparing, sorting, and finding the index of a substring. Some of these are culture-specific, and some aren't. For instance, in all cultures (as far as I can see), "lassen" and "la\u00dfen" (a "sharp S" or eszett being the Unicode-escaped character in there) are considered equal when CompareTo or Compare are used, but not when Equals is used. IndexOf will treat the eszett as the same as "ss", unless you use a CompareInfo.IndexOf and specify CompareOptions.Ordinal as the options to use.

Some other unicode character appear to be completely invisible to the normal IndexOf. Someone asked in the C# newsgroup why a search/replace method was going into an infinite loop. It was repeatedly using Replace to replace all double spaces with a single space, and checking whether or not it had finished by using IndexOf, so that multiple spaces would collapse to a single space. Unfortunately, this was failing due to a "strange" character in the original string between two spaces. IndexOf matched the double space, ignoring the extra character, but Replace didn't. I don't know which exact character was in the real data, but it can be easily reproduced using U+200C which is a zero-width non-joiner character (whatever that means, exactly!). Put one of those in the middle of the text you're searching in, and IndexOf will ignore it, but Replace won't. Again, to make the two methods behave the same, you can use CompareInfo.IndexOf and pass in CompareOptions.Ordinal. My guess is that there's a lot of code which would fail on "awkward" data like this. (I wouldn't for a moment claim that all my code is immune, either.)

Constructors in C#

2008-01-10T06:16:00.001-08:00

In the C# newsgroup, there was recently (at the time of writing) a discussion about various aspects of constructors. This page provides most of the important bits about how I believe constructors work in C#, including references to the language specification to back them up. If you disagree with anything in this page, I ask you to follow the language specification link to the appropriate section. If you then believe I'm incorrectly interpreting the spec, please mail me with details. The ECMA spec and the MS spec are identical as far as I've seen, apart from how they number their sections. I'll give links to both so you can use whichever you feel more comfortable with. In fact,

A little terminology...
The C# language spec refers to two types of constructors: instance constructors and static constructors. This page only deals with instance constructors, and I'll just call them constructors for short. That's what most people understand by the term "constructor" anyway. Static constructors are the equivalent of static initializers in Java - they give the code which is executed when a class is first used.

What is a constructor?
(ECMA) (MS).
To quote the spec, "An instance constructor is a member that implements the actions required to initialize an instance of a class." That's actually a pretty good description on its own. A constructor is invoked when you use the "new" operator, or use the various methods of reflection to create an instance of a class.

What do constructors look like?
A constructor looks very much like a method, but with no return type and a name which is the same as the name of the class. (The modifiers you can use with a constructor are slightly different, however - see the spec for more information about that.) Here's a short class to use as an example:

public class MySimpleClass
{
public MySimpleClass (int x)
{
Console.WriteLine (x);
}
}

This class has a single constructor, which takes an int as a parameter, and prints that int's value to the console.

Constructor Initializers
Now, in fact, there's something else going on here, which is hidden by the compiler. The above is actually equivalent to:

public class MySimpleClass
{
public MySimpleClass (int x) : base()
{
Console.WriteLine (x);
}
}

(ECMA) (MS).
Note the base() bit. That's saying which other constructor should be invoked before any of the rest of the code in the constructor runs. Every constructor of every class other than plain object does this, either explicitly or implicitly. There are two forms of constructor initializer - one which calls a base class constructor (as above) and one which calls another constructor from this class, using the this (...) syntax. There must always be a "chain" of constructors which runs constructors all the way up the class hierarchy. Every class in the hierarchy will have a constructor invoked, although some of those constructors may not explicitly appear in the code. (See the section on default constructors, later.) The parameters (if any) within the brackets of base(...) or this(...) are passed as the parameters to the invoked constructors. They can be the parameters given in the constructor declaration, but don't have to be. Here's an example:

public class MyBaseClass
{
public MyBaseClass (int x) : base() // Invoke the parameterless constructor in object
{
Console.WriteLine ("In the base class constructor taking an int, which is "+x);
}
}

public class MyDerivedClass : MyBaseClass
{
public MyDerivedClass () : this (5) // Invoke the MyDerivedClass constructor taking an int
{
Console.WriteLine ("In the derived class parameterless constructor.");
}

public MyDerivedClass (int y) : base (y+1) // Invoke the MyBaseClass constructor taking an int
{
Console.WriteLine ("In the derived class constructor taking an int parameter.");
}

public MyDerivedClass (string x) : base (10) // Invoke the MyBaseClass constructor taking an int
{
Console.WriteLine ("In the derived class constructor taking a string parameter.");
}
}

With the above code, a bit of code saying new MyDerivedClass(); would invoke the MyDerivedClass parameterless constructor, which would in turn invoke the MyDerivedClass constructor which takes an int parameter (with 5 as that parameter value), which would in turn invoke the MyBaseClass constructor which takes an int parameter (with 6 as that parameter value). Note that the specified constructor is run before the constructor body, so the result of new MyDerivedClass(); would be the following output on the console:

In the base class constructor taking an int, which is 6
In the derived class constructor taking an int parameter.
In the derived class parameterless constructor.

Not all constructors in the hierarchy need to be invoked (as demonstrated above - the constructor taking a string parameter is not invoked at all when you do new MyDerivedClass();) but as I said earlier, there must be at least one constructor invoked in each class in the hierarchy.

Default constructor initializers
(ECMA) (MS), second paragraph.
Any constructor which doesn't have a constructor initializer has one provided for it by the compiler. The initializer is of the form base() - in other words, a call to the base class's parameterless constructor. You'll get a compile-time error if you don't provide a constructor initializer and the base class doesn't have an accessible parameterless constructor. (Note, however, that it may have one without you putting one in yourself - see the next section.)

Default constructors
(ECMA) (MS)
If you don't specify any constructors at all, a default constructor is provided by the compiler. This default constructor is a parameterless constructor with no body, which calls the parameterless constructor of the base class. In other words:

public class MySimpleClass
{
int someMemberVariable;
}

... is exactly equivalent to:

public class MySimpleClass
{
int someMemberVariable;

public MySimpleClass() : base()
{
}
}

Following from the previous section, this means that if the base class has no accessible parameterless constructor (including a default one), you get a compile-time error if the derived class doesn't have any constructors - because the default constructor will implicitly try to call a parameterless constructor from the base type.

If the class is abstract, the default constructor provided has protected accessibility; otherwise it has public accessibility.

Instance variable initializers
(ECMA) (MS)
When a member variable declaration also has an assignment, that's called a variable initializer. All the variable initializers for a class are implicitly run directly before the invocation of whichever base class constructor is invoked. (Note that this is a change from a previous version of the page, where I believed that they were invoked after the base constructor, as they are in Java.) Here's a simple example of instance variable initializers:

public class MySimpleClass
{
int someMemberVariable = 10;

public MySimpleClass()
{
Console.WriteLine ("someMemberVariable={0}", someMemberVariable);
}
}

The output of the above is 10 when a new instance is created, whereas without the instance variable initializer, it would be 0 (the default value for instance variables of type int). Demonstrating the difference between Java and C# in terms of when the instance variable initializers are run requires calling an overridden method from the base class constructor. This is a really bad idea - wherever possible, only call non-virtual methods from constructors, and if you absolutely must call a virtual method, document very carefully that it is called from the constructor, so that people wishing to override it are aware that their object may not be in a consistent state when it is called (as their own constructor won't have run yet). Here's an example:

public class MyBaseClass
{
public MyBaseClass ()
{
Console.WriteLine (this.ToString());
}
}

public class MyDerivedClass : MyBaseClass
{
string name="hello";

public MyDerivedClass : base()
{
Console.WriteLine (this.ToString());
}

public override string ToString()
{
return name;
}
}

When a new instance of MyDerivedClass is created in C#, the output is:

hello
hello

The first line is hello because the instance variable initializer for the name variable has run directly before the base class constructor. The equivalent code in Java syntax would output:

null
hello

Here the first line is null because the instance variable initializer for the name variable only runs directly after the base class constructor has returned (but before ToString is called for the second time).

Constructors are not inherited
(ECMA) (MS), second paragraph from the bottom.
Constructors are not inherited. In other words, just because a base class has a constructor taking a certain list of parameters doesn't mean that a derived class has a constructor taking that list of parameters. (It can, by providing one itself, but it doesn't inherit it from the base class.) To demonstrate this, here's an example which doesn't compile:

public class MyBaseClass
{
public MyBaseClass (int x)
{
}
}

public class MyDerivedClass : MyBaseClass
{
// This constructor itself is okay - it invokes an
// appropriate base class constructor
public MyDerivedClass () : base (5)
{
}

public static void Main()
{
new MyDerivedClass (10);
}
}

Here, we try to invoke a constructor for MyDerivedClass which takes an int parameter. There isn't one, however, as constructors aren't inherited. The MyBaseClass constructor which takes an int parameter can be invoked by a constructor in MyDerivedClass (as is shown by the parameterless MyDerivedClass constructor) but isn't actually inherited. Removing the "10" from the above code would make it compile and run with no problems - the parameterless MyDerivedClass constructor would be invoked, and that would in turn invoke the MyBaseClass constructor taking an int parameter, with 5 as that parameter value.

Some people have said that they would rather constructors were inherited, making the language act as if all derived classes had constructors with all the parameter lists from the constructors from the base class, and just invoking them with the parameters provided. I believe this would be a very bad idea. Take, for instance, the FileInfo class. You must logically provide a filename when constructing a FileInfo instance, as otherwise it won't know what it's meant to be providing information on. However, as object has a parameterless constructor, constructors being inherited would then mean that FileInfo had a parameterless constructor. Some have suggested that this could be fixed by allowing you to "override" the parameters you didn't want invoked as private, but this goes against the idea that you should never be able to override anything to give it more restrictive access, and also means that class developers would have to change their code every time a new constructor was added to a base class.

Delegates and events

2008-01-10T06:14:00.000-08:00

People often find it difficult to see the difference between events and delegates. C# doesn't help matters by allowing you to declare field-like events which are automatically backed by a delegate variable of the same name. This article aims to clarify the matter for you. Another source of confusion is the overloading of the term "delegate". Sometimes it is used to mean a delegate type, and at other times it can be used to mean an instance of a delegate type. I'll use "delegate type" and "delegate instance" to distinguish between them, and "delegate" when talking about the whole topic in a general sense.

Delegate types
In some ways, you can think of a delegate type as being a bit like an interface with a single method. It specifies the signature of a method, and when you have a delegate instance, you can make a call to it as if it were a method with the same signature. Delegates provide other features, but the ability to make calls with a particular signature is the reason for the existence of the delegate concept. Delegates hold a reference to a method, and (for instance methods) a reference to the target object the method should be called on.

Delegates types are declared with the delegate keyword. They can appear either on their own or nested within a class, as shown below.

namespace DelegateArticle
{
public delegate string FirstDelegate (int x);

public class Sample
{
public delegate void SecondDelegate (char a, char b);
}
}

This code declares two delegate types. The first is DelegateArticle.FirstDelegate which has a single parameter of type int and returns a string. The second is DelegateArticle.Sample.SecondDelegate which has two char parameters, and doesn't return anything (because the return type is specified as void).

Note that the delegate keyword doesn't always mean that a delegate type is being declared. As we'll see in a moment, the same keyword is used when creating instances of the delegate type.

The types declared here derive from System.MulticastDelegate, which in turn derives from System.Delegate. In practice, you'll only see delegate types deriving from MulticastDelegate. The difference between Delegate and MulticastDelegate is largely historical; in betas of .NET 1.0 the difference was significant (and annoying) - Microsoft considered merging the two types together, but decided it was too late in the release cycle to make such a major change. You can pretty much pretend that they're only one type.

Any delegate type you create has the members inherited from its parent types, one constructor with parameters of object and IntPtr and three extra methods: Invoke, BeginInvoke and EndInvoke. We'll come back to the constructor in a minute. The methods can't be inherited from anything, because the signatures vary according to the signature the delegate is declared with. Using the sample code above, the first delegate has the following methods:

public string Invoke (int x);
public System.IAsyncResult BeginInvoke(int x, System.AsyncCallback callback, object state);
public string Endinvoke(IAsyncResult result);

As you can see, the return type of Invoke and EndInvoke matches that of the declaration signature, as are the parameters of Invoke and the first parameters of BeginInvoke. We'll see the purpose of Invoke in the next section, and cover BeginInvoke and EndInvoke in the section on advanced usage. It's a bit premature to talk about calling methods when we don't know how to create an instance, however. We'll cover that (and more) in the next section.

Delegate instances: the basics
Now we know how a delegate type is declared and what it contains, let's look at how to create an instance of such a type, and what we can do with it.

Creating delegate instances
Note: this article doesn't cover the features of C# 2.0 and 3.0 for creating delegate instances. I cover the C# 2.0 features in my article about C# 2.0. When I eventually get round to writing about C# 3.0, I'm sure I'll cover the new features for lambda functions etc at that point. By concentrating on the explicit manner of creating instances in C# 1.0/1.1, I believe it will be easier to understand what's going on under the hood. When you understand the basics, it's clearly worth knowing the features these later versions provide - but if you try to use them without having a firm grasp on the basics, you may well get confused.

As mentioned earlier, the key points of data in any particular delegate instance are the method the delegate refers to, and a reference to call the method on (the target). For static methods, no target is required. The CLR itself supports other slightly different forms of delegate, where either the first argument passed to a static method is held within the delegate, or the target of an instance method is provided as an argument when the method is called. See the documentation for System.Delegate for more information on this if you're interested, but don't worry too much about it - it's largely irrelevant from a C# point of view. C# only supports creating what MSDN calls a closed instance delegate type.

So, now that we know the two pieces of data required to create an instance (along with the type itself, of course), how do we tell the compiler what they are? We use what the C# specification calls a delegate-creation-expression which is of the form new delegate-type (expression). The expression must either be another delegate of the same type (or a compatible delegate type in C# 2.0) or a method group - the name of a method and optionally a target, specified as if you were calling the method, but without the arguments or brackets. Creating copies of a delegate is fairly rare, so we will concentrate on the more common form. A few examples are listed below:

// The following two creation expressions are equivalent,
// where InstanceMethod is an instance method in the class
// containing the creation expression (or a base class).
// The target is "this".
FirstDelegate d1 = new FirstDelegate(InstanceMethod);
FirstDelegate d2 = new FirstDelegate(this.InstanceMethod);

// Here we create a delegate instance referring to the same method
// as the first two examples, but with a different target.
FirstDelegate d3 = new FirstDelegate(anotherInstance.InstanceMethod);

// This delegate instance uses an instance method in a different class,
// specifying the target to call the method on
FirstDelegate d4 = new FirstDelegate(instanceOfOtherClass.OtherInstanceMethod);

// This delegate instance uses a static method in ths class containing
// the creation expression (or a base class).
FirstDelegate d5 = new FirstDelegate(StaticMethod);

// This delegate instance uses a static method in a different class
FirstDelegate d6 = new FirstDelegate(OtherClass.OtherStaticMethod);

The constructor we mentioned earlier has two parameters - an object and an IntPtr. The object is a reference to the target (or null for static methods) and the IntPtr is a pointer to the method itself.

One point to note is that delegate instances can refer to methods and targets which wouldn't normally be visible at the point the call is actually made. For instance, a private method can be used to create a delegate instance, and then the delegate instance can be returned from a public member. Alternatively, the target of an instance may be an object which the eventual caller knows nothing about. However, both the target and the method must be accessible to the creating code. In other words, if (and only if) you can call a particular method on a particular object, you can use that method and target for delegate creation. Access rights are effectively ignored at call time. Speaking of which...

Calling delegate instances
Delegate instances are called just as if they were the methods themselves. For instance, to call the delegate referred to by variable d1 above, we could write:

string result = d1(10);

The method referred to by the delegate instance is called on the target object (if there is one), and the result is returned. Producing a complete program to demonstrate this without including a lot of seemingly irrelevant code is tricky. However, here's a program which gives one example of a static method and one of an instance method. DelegateTest.StaticMethod could be written as just StaticMethod in the same way that (within an instance method) you could write InstanceMethod instead of this.InstanceMethod - I've included the class name just to make it clear how you would reference methods from other classes.

using System;

public delegate string FirstDelegate (int x);

class DelegateTest
{
string name;

static void Main()
{
FirstDelegate d1 = new FirstDelegate(DelegateTest.StaticMethod);

DelegateTest instance = new DelegateTest();
instance.name = "My instance";
FirstDelegate d2 = new FirstDelegate(instance.InstanceMethod);

Console.WriteLine (d1(10)); // Writes out "Static method: 10"
Console.WriteLine (d2(5)); // Writes out "My instance: 5"
}

static string StaticMethod (int i)
{
return string.Format ("Static method: {0}", i);
}

string InstanceMethod (int i)
{
return string.Format ("{0}: {1}", name, i);
}
}

The C# syntax is just a short-hand for calling the Invoke method provided by each delegate type. Delegates can also be run asynchronously if they provide BeginInvoke/EndInvoke methods. These are explained later.

Combining delegates
Delegates can be combined such that when you call the delegate, a whole list of methods are called - potentially with different targets. When I said before that a delegate contained a target and a method, that was a slight simplification. That's what a delegate instance representing one method contains. For the sake of clarity, I'll refer to such delegate instances as simple delegates. The alternative is a delegate instance which is effectively a list of simple delegates, all of the same type (i.e. having the same signature). I'll call these combined delegates. Combined delegates can themselves be combined together, effectively creating one big list of simple delegates in the obvious fashion.

It's important to understand that delegate instances are always immutable. Anything which combines them together (or takes one away from the other) creates a new delegate instance to represent the new list of targets/methods to call. This is just like strings: if you call String.PadLeft for instance, it doesn't actually change the string you call it on - it just returns a new string with the appropriate padding.

Combining two delegate instances is usually done using the addition operator, as if the delegate instances were strings or numbers. Subtracting one from another is usually done with the subtraction operator. Note that when you subtract one combined delegate from another, the subtraction works in terms of lists. If the list to subtract is not found in the original list, the result is just the original list. Otherwise, the last occurrence of the list is removed. This is best shown with some examples. Instead of actual code, the following uses lists of simple delegates d1, d2 etc. For instance, [d1, d2, d3] is a combined delegate which, when executed, would call d1 then d2 then d3. An empty list is represented by null rather than an actual delegate instance.

Expression Result
null + d1 d1
d1 + null d1
d1 + d2 [d1, d2]
d1 + [d2, d3] [d1, d2, d3]
[d1, d2] + [d2, d3] [d1, d2, d2, d3]
[d1, d2] - d1 d2
[d1, d2] - d2 d1
[d1, d2, d1] - d1 [d1, d2]
[d1, d2, d3] - [d1, d2] d3
[d1, d2, d3] - [d2, d1] [d1, d2, d3]
[d1, d2, d3, d1, d2] - [d1, d2] [d1, d2, d3]
[d1, d2] - [d1, d2] null

Delegate instances can also be combined with the static Delegate.Combine method, and one can be subtracted from another with the static Delegate.Remove method. The C# compiler converts the addition and subtraction operators into calls to these methods. Because they are static methods, they work easily with null references.

The addition and subtraction operators always work as part of assignment: d1 += d2; is exactly equivalent to d1 = d1+d2;, and likewise for subtraction. Again, the original delegate instance remains unchanged; the value of d1 just changes to be a reference to the appropriate new combined delegate.

Note that because extra delegates are both added to and removed from the end of the list, x += y; x -= y; is always a no-op.

If a delegate type is declared to return a value (i.e. it's not declared with a void return type) and a combined delegate instance is called, the value returned from that call is the one returned by the last simple delegate in the list.

Events
First things first: events aren't delegate instances. Let's try that again.

Events aren't delegate instances.

It's unfortunate in some ways that C# lets you use them in the same way in certain situations, but it's very important that you understand the difference.

I find the easiest way to understand events is to think of them a bit like properties. While properties look like they're fields, they're definitely not - and you can write properties which don't use fields at all. Similarly, while events look like delegate instances in terms of the way you express the add and remove operations, they're not.

Events are pairs of methods, appropriately decorated in IL to tie them together and let languages know that the methods represent events. The methods correspond to add and remove operations, each of which take a delegate instance parameter of the same type (the type of the event). What you do with those operations is pretty much up to you, but the typical use is to add or remove the delegate from a list of handlers for the event. When the event is triggered (whatever that trigger might be - a button click, a timeout, an unhandled exception) the handlers are called in turn. Note that in C#, the calling of the event handlers is not part of the event itself. (CIL defines an association with a raise_eventName method, and indeed "other" methods, but these are not used in C#.)

The add and remove methods are called in C# using eventName += delegateInstance; and eventName -= delegateInstance; respectively, where eventName may be qualified with a reference (e.g. myForm.Click) or a type name (e.g. MyClass.SomeEvent). Static events are relatively rare.

Events themselves can be declared in two ways. The first is with explicit add and remove methods, declared in a very similar way to properties, but with the event keyword. Here's an example of an event for the System.EventHandler delegate type. Note how it doesn't actually do anything with the delegate instances which are passed to the add and remove methods - it just prints out which operation has been called. Note that the remove operation is called even though we've told it to remove null.

using System;

class Test
{
public event EventHandler MyEvent
{
add
{
Console.WriteLine ("add operation");
}

remove
{
Console.WriteLine ("remove operation");
}
}

static void Main()
{
Test t = new Test();

t.MyEvent += new EventHandler (t.DoNothing);
t.MyEvent -= null;
}

void DoNothing (object sender, EventArgs e)
{
}
}

Although it would be very rare to ignore the value in this way, there are times when you don't want to back an event with a simple delegate variable. For instance, in situations where there are lots of events but only a few are likely to be subscribed to, you could have a map from some key describing the event to the delegate currently handling it. This is what Windows Forms does - it means that you can have a huge number of events without wasting a lot of memory with variables which will usually just have null values.

A shortcut: field-like events
C# provides a simple way of declaring both a delegate variable and an event at the same time. This is called a field-like event, and is declared very simply - it's the same as the "longhand" event declaration, but without the "body" part:

public event EventHandler MyEvent

This creates a delegate variable and an event, both with the same type. The access to the event is determined by the event declaration (so the example above creates a public event, for instance) but the delegate variable is always private. The implicit body of the event is the obvious one to add/remove delegate instances to the delegate variable, but the changes are made within a lock. For C# 1.1, the event is equivalent to:

private EventHandler _myEvent;

public event EventHandler MyEvent
{
add
{
lock (this)
{
_myEvent += value;
}
}
remove
{
lock (this)
{
_myEvent -= value;
}
}
}

That's for an instance member. For an event declared as static, the variable is also static and a lock is taken out on typeof(XXX) where XXX is the name of the class declaring the event. In C# 2.0 there is little guarantee about what is used for locking - only that a single object associated with the instance is used for locking with instance events, and a single object associated with the class is used for locking static events. (Note that this only holds for class events, not struct events - there are issues in terms of locking with struct events; in practice I don't remember ever seeing a struct with any events.) None of this is actually as useful as you might think - see the threading section for more details.

So, what happens when you refer to MyEvent in code? Well, within the text of type itself (including nested types) the compiler generates code which refers to the delegate variable (_myEvent in my sample above). In all other contexts, the compiler generates code which refers to the event.

What's the Point?
Now we know what they are, what's the point of having both delegates and events? The answer is encapsulation. Suppose events didn't exist as a concept in C#/.NET. How would another class subscribe to an event. Three options:

A public delegate variable
A delegate variable backed by a property
A delegate variable with AddXXXHandler and RemoveXXXHandler methods
Option 1 is clearly horrible, for all the normal reasons we abhor public variables. Option 2 is slightly better, but allows subscribers to effectively override each other - it would be all too easy to write someInstance.MyEvent = eventHandler; which would replace any existing event handlers rather than adding a new one. In addition, you still need to write the properties.

Option 3 is basically what events give you, but with a guaranteed convention (generated by the compiler and backed by extra flags in the IL) and a "free" implementation if you're happy with the semantics that field-like events give you. Subscribing to and unsubscribing from events is encapsulated without allowing arbitrary access to the list of event handlers, and languages can make things simpler by providing syntax for both declaration and subscription.

Thread-safe events
Earlier we touched on field-like events locking during the add/remove operations. This is to provide a certain amount of thread safety. Unfortunately, it's not terribly useful. Firstly, even with 2.0, the spec allows for the lock to be the reference to this object, or the type itself for static events. That goes against the principle of locking on privately held references to avoid accidental deadlocks.

Ironically, the second problem is the exact reverse of the first - because in C# 2.0 you can't guarantee which lock is going to be used, you can't use it yourself when raising an event to ensure that you see the most recent value in the thread doing the raising. You can lock on something else or call one of the memory barrier methods, but it leaves something of a nasty taste in the mouth.

If you want to be truly thread-safe, such that when you raise an event you always use the most recent value of the delegate variable, along with making sure that the add/remove operations don't interfere with each other, you need to write the body of the add/remove operations yourself. Here's an example:

///

/// Delegate variable backing the SomeEvent event.
///

SomeEventHandler someEvent;

///

/// Lock for SomeEvent delegate access.
///

readonly object someEventLock = new object();

///

/// Description for the event
///

public event SomeEventHandler SomeEvent
{
add
{
lock (someEventLock)
{
someEvent += value;
}
}
remove
{
lock (someEventLock)
{
someEvent -= value;
}
}
}

///

/// Raises the SomeEvent event
///

protected virtual OnSomeEvent(EventArgs e)
{
SomeEventHandler handler;
lock (someEventLock)
{
handler = someEvent;
}
if (handler != null)
{
handler (this, e);
}
}

You could use a single lock for all your events, or even for other things as well - it depends on your situation. Note that you need to assign the current value to a local variable inside the lock (to get the most recent value) and then test it for nullity and execute it outside the lock: holding the lock whilst raising the event is a very bad idea, as you could easily deadlock. (Event handlers may well need to wait for another thread to do something, and if that other thread were to call the add or remove operation on your event, you'd get deadlock.)

This all works because once handler has been assigned the value of someEvent, the value of handler won't change even if someEvent does. So if all the handlers are unsubscribed from the event, someEvent will become null but handler will still have whatever value it had when it was assigned. In fact, as delegate instances are immutable, whatever handlers were subscribed when the handler = someEvent; line was executed will be called, even if others have subscribed between then and the handler (this, e); line.

Now, it's important to consider whether or not you even need thread safety. Are event handlers going to be added or removed from other threads? Do you need to raise the event from another thread? If you're in complete control of your application, the answer may very well be "no". (If you're writing a class library, it's more likely that being thread-safe is important.) If you don't need thread safety, you may want to implement the add/remove operations to get round the problem of the externally visible lock that C# uses (or may use in the case of 2.0). At that point, the operations become pretty trivial. Here's the equivalent of the earlier code, but without thread safety.

///

/// Delegate variable backing the SomeEvent event.
///

SomeEventHandler someEvent;

///

/// Description for the event
///

public event SomeEventHandler SomeEvent
{
add
{
someEvent += value;
}
remove
{
someEvent -= value;
}
}

///

/// Raises the SomeEvent event
///

protected virtual OnSomeEvent(EventArgs e)
{
if (someEvent != null)
{
someEvent (this, e);
}
}

The check for nullity is due to delegate variables being null when there aren't any delegate instances to call. One way to make things simpler is to use a no-op delegate instance as the "default" one, which is never removed. At that point, you can just obtain the value of the delegate variable (inside a lock if you're being thread-safe) and then execute the delegate instance. If there are no "real" delegate targets to call, the no-op target will execute and that's all that will happen.

Delegate instances: other methods
Earlier we saw how a call to someDelegate(10) is actually a short-hand for someDelegate.Invoke(10). Delegates types may also allow asynchronous behaviour using the BeginInvoke/EndInvoke pair. These are optional as far as the CLI specification is concerned, but C# delegate types always provide them. They follow the same model for asynchronous execution as the rest of .NET, allowing a callback handler to be provided, along with an object to store state information. The delegates are executed on threads created by the system thread-pool.

The first example below operates without a callback, simply using BeginInvoke and EndInvoke from the same thread. This is occasionally useful when a single thread is used for an operation which is synchronous in general, but which contains elements which may be performed in parallel. The methods involved are all static for the sake of simplicity, but delegate instances with specific target objects can also be used, and often are. EndInvoke returns whatever value was returned by the delegate call. If the call threw an exception, the same exception is thrown by EndInvoke.

using System;
using System.Threading;

delegate int SampleDelegate(string data);

class AsyncDelegateExample1
{
static void Main()
{
SampleDelegate counter = new SampleDelegate(CountCharacters);
SampleDelegate parser = new SampleDelegate(Parse);

IAsyncResult counterResult = counter.BeginInvoke ("hello", null, null);
IAsyncResult parserResult = parser.BeginInvoke ("10", null, null);
Console.WriteLine ("Main thread continuing");

Console.WriteLine ("Counter returned {0}", counter.EndInvoke(counterResult));
Console.WriteLine ("Parser returned {0}", parser.EndInvoke(parserResult));

Console.WriteLine ("Done");
}

static int CountCharacters (string text)
{
Thread.Sleep (2000);
Console.WriteLine ("Counting characters in {0}", text);
return text.Length;
}

static int Parse (string text)
{
Thread.Sleep (100);
Console.WriteLine ("Parsing text {0}", text);
return int.Parse(text);
}
}

The calls to Thread.Sleep are just to demonstrate that the execution really does occur in parallel. The sleep in CountCharacters is as large as it is to force the system thread-pool to run the tasks on two different threads - the thread-pool serializes requests which don't take long in order to avoid creating more threads than it needs to. By sleeping for a long time, we're simulating a long-running request. Here's the output from a sample run:

Main thread continuing
Parsing text 10
Counting characters in hello
Counter returned 5
Parser returned 10
Done

The calls to EndInvoke block until the delegate has completed in much the same way as calls to Thread.Join block until the threads involved have terminated. The IAsyncResult values returned by the calls to BeginInvoke allows access to the state passed as the last parameter to BeginInvoke, but this isn't typically used in the style of asynchronous invocation shown above.

The code above is fairly simple, but often not as powerful as a model which uses callbacks after the delegate has completed. Typically, the callback will call EndInvoke to obtain the result of the delegate. Although it is still a theoretically blocking call, it will never actually block because the callback only executes when the delegate has completed anyway. The callback may well use the state provided to BeginInvoke as extra context information. The sample code below uses the same counting and parsing delegates as the previous example, but with a callback displaying the results. The state is used to determine how to format each result, so a single callback can be used for both asynchronous calls. Note the cast from IAsyncResult to AsyncResult: the value provided to the callback is always an instance of AsyncResult, and this can be used to obtain the original delegate instance, so that the callback can call EndInvoke. It is somewhat anomalous that AsyncResult lives in the System.Runtime.Remoting.Messaging namespace when all the other classes involved are in either System or System.Threading, but such is life.

using System;
using System.Threading;
using System.Runtime.Remoting.Messaging;

delegate int SampleDelegate(string data);

class AsyncDelegateExample2
{
static void Main()
{
SampleDelegate counter = new SampleDelegate(CountCharacters);
SampleDelegate parser = new SampleDelegate(Parse);

AsyncCallback callback = new AsyncCallback (DisplayResult);

counter.BeginInvoke ("hello", callback, "Counter returned {0}");
parser.BeginInvoke ("10", callback, "Parser returned {0}");

Console.WriteLine ("Main thread continuing");

Thread.Sleep (3000);
Console.WriteLine ("Done");
}

static void DisplayResult(IAsyncResult result)
{
string format = (string) result.AsyncState;
AsyncResult delegateResult = (AsyncResult) result;
SampleDelegate delegateInstance = (SampleDelegate)delegateResult.AsyncDelegate;

Console.WriteLine (format, delegateInstance.EndInvoke(result));
}

static int CountCharacters (string text)
{
Thread.Sleep (2000);
Console.WriteLine ("Counting characters in {0}", text);
return text.Length;
}

static int Parse (string text)
{
Thread.Sleep (100);
Console.WriteLine ("Parsing text {0}", text);
return int.Parse(text);
}
}

This time almost all the work is done on thread-pool threads. The main thread just kicks off the asynchronous tasks and then sleeps for long enough to let all the work finish. (Thread-pool threads are background threads - without the extra Sleep call, the application would terminate before the delegate calls finished executing.) Some sample output is below - notice how this time, because there is no guaranteed ordering to the calls to EndInvoke, the parser result is displayed before the counter result. In the previous example, the parser almost certainly completed before the counter did, but the main thread waited to obtain the result of the counter first.

Main thread continuing
Parsing text 10
Parser returned 10
Counting characters in hello
Counter returned 5
Done

Note that you must call EndInvoke when you use asynchronous execution in order to guarantee not to leak memory or handles. Some implementations may not leak, but you shouldn't rely on this. See my thread-pool article for some sample code to allow "fire and forget" style asynchronous behaviour if this is inconvenient.

Parameter passing in C#

2008-01-10T06:10:00.000-08:00

Many people have become fairly confused about how parameters are passed in C#, particularly with regard to reference types. This page should help to clear up some of that confusion. If you have any suggestions for how it can be made clearer, please mail me.

Microsoft also has a good page about this topic (which I believe uses exactly the same terminology as this page - let me know if they appear to disagree).

Note: Lee Richardson has written a complementary article to this one, particularly for those who learn well with pictures. Basically it illustrates the same points, but using pretty diagrams to show what's going on.

Table of contents
Preamble: what is a reference type?
Further preamble: what is a value type?
Checking you understand the preamble...
The different kinds of parameters
Value parameters
Reference parameters
Output parameters
Parameter arrays
Mini-glossary
Preamble: what is a reference type?
In .NET (and therefore C#) there are two main sorts of type: reference types and value types. They act differently, and a lot of confusion about parameter passing is really down to people not properly understanding the difference between them. Here's a quick explanation:

A reference type is a type which has as its value a reference to the appropriate data rather than the data itself. For instance, consider the following code:

StringBuilder sb = new StringBuilder();

(I have used StringBuilder as a random example of a reference type - there's nothing special about it.) Here, we declare a variable sb, create a new StringBuilder object, and assign to sb a reference to the object. The value of sb is not the object itself, it's the reference. Assignment involving reference types is simple - the value which is assigned is the value of the expression/variable - i.e. the reference. This is demonstrated further in this example:

StringBuilder first = new StringBuilder();
StringBuilder second = first;

Here we declare a variable first, create a new StringBuilder object, and assign to first a reference to the object. We then assign to second the value of first. This means that they both refer to the same object. They are still, however, independent variables themselves. Changing the value of first will not change the value of second - although while their values are still references to the same object, any changes made to the object through the first variable will be visible through the second variable. Here's a demonstration of that:

StringBuilder first = new StringBuilder();
StringBuilder second = first;
first.Append ("hello");
first = null;
Console.WriteLine (second);

(Download sample code) Output:

hello

Here, we declare a variable first, create a new StringBuilder object, and assign to first a reference to the object. We then assign to second the value of first. We then call the Append method on this object via the reference held in the first variable. After this, we set the first variable to null (a value which doesn't refer to any object). Finally, we print out the results of calling the ToString method on the StringBuilder object via the reference held in the second variable. hello is displayed, demonstrating that even though the value of first has changed, the data within the object it used to refer to hasn't - and second still refers to that object.

Class types, interface types, delegate types and array types are all reference types.

Further preamble: what is a value type?
While reference types have a layer of indirection between the variable and the real data, value types don't. Variables of a value type directly contain the data. Assignment of a value type involves the actual data being copied. Take a simple struct, for example:

public struct IntHolder
{
public int i;
}

Wherever there is a variable of type IntHolder, the value of that variable contains all the data - in this case, the single integer value. An assignment copies the value, as demonstrated here:

IntHolder first = new IntHolder();
first.i=5;
IntHolder second = first;
first.i=6;
Console.WriteLine (second.i);

(Download sample code) Output:

5

Here, second.i has the value 5, because that's the value first.i has when the assignment second=first occurs - the values in second are independent of the values in first apart from when the assignment takes place.

Simple types (such as float, int, char), enum types and struct types are all value types.

Note that many types (such as string) appear in some ways to be value types, but in fact are reference types. These are known as immutable types. This means that once an instance has been constructed, it can't be changed. This allows a reference type to act similarly to a value type in some ways - in particular, if you hold a reference to an immutable object, you can feel comfortable in returning it from a method or passing it to another method, safe in the knowledge that it won't be changed behind your back. This is why, for instance, the string.Replace doesn't change the string it is called on, but returns a new instance with the new string data in - if the original string were changed, any other variables holding a reference to the string would see the change, which is very rarely what is desired.

Constrast this with a mutable (changeable) type such as ArrayList - if a method returns the ArrayList reference stored in an instance variable, the calling code could then add items to the list without the instance having any say about it, which is usually a problem. Having said that immutable reference types act like value types, they are not value types, and shouldn't be thought of as actually being value types.

For more information about value types, reference types, and where the data for each is stored in memory, please see my other article about the subject.

Checking you understand the preamble...
What would you expect to see from the code above if the declaration of the IntHolder type was as a class instead of a struct? If you don't understand why the output would be 6, please re-read both preambles and mail me if it's still not clear - if you don't get it, it's my fault, not yours, and I need to improve this page. If you do understand it, parameter passing becomes very easy to understand - read on.

The different kinds of parameters
There are four different kinds of parameters in C#: value parameters (the default), reference parameters (which use the ref modifier), output parameters (which use the out modifier), and parameter arrays (which use the params modifier). You can use any of them with both value and reference types. When you hear the words "reference" or "value" used (or use them yourself) you should be very clear in your own mind whether you mean that a parameter is a reference or value parameter, or whether you mean that the type involved is a reference or value type. If you can keep the two ideas separated, they're very simple.

Value parameters
By default, parameters are value parameters. This means that a new storage location is created for the variable in the function member declaration, and it starts off with the value that you specify in the function member invocation. If you change that value, that doesn't alter any variables involved in the invocation. For instance, if we have:

void Foo (StringBuilder x)
{
x = null;
}

...

StringBuilder y = new StringBuilder();
y.Append ("hello");
Foo (y);
Console.WriteLine (y==null);

(Download sample code) Output:

False

The value of y isn't changed just because x is set to null. Remember though that the value of a reference type variable is the reference - if two reference type variables refer to the same object, then changes to the data in that object will be seen via both variables. For example:

void Foo (StringBuilder x)
{
x.Append (" world");
}

...

StringBuilder y = new StringBuilder();
y.Append ("hello");
Foo (y);
Console.WriteLine (y);

(Download sample code) Output:

hello world

After calling Foo, the StringBuilder object that y refers to contains "hello world", as in Foo the data " world" was appended to that object via the reference held in x.

Now consider what happens when value types are passed by value. As I said before, the value of a value type variable is the data itself. Using the previous definition of the struct IntHolder, let's write some code similar to the above:

void Foo (IntHolder x)
{
x.i=10;
}

...

IntHolder y = new IntHolder();
y.i=5;
Foo (y);
Console.WriteLine (y.i);

(Download sample code) Output:

5

When Foo is called, x starts off as a struct with value i=5. Its i value is then changed to 10. Foo knows nothing about the variable y, and after the method completes, the value in y will be exactly the same as it was before (i.e. 5).

As we did earlier, check that you understand what would happen if IntHolder was declared as a class instead of a struct. You should understand why y.i would be 10 after calling Foo in that case.

Reference parameters
Reference parameters don't pass the values of the variables used in the function member invocation - they use the variables themselves. Rather than creating a new storage location for the variable in the function member declaration, the same storage location is used, so the value of the variable in the function member and the value of the reference parameter will always be the same. Reference parameters need the ref modifier as part of both the declaration and the invocation - that means it's always clear when you're passing something by reference. Let's look at our previous examples, just changing the parameter to be a reference parameter:

void Foo (ref StringBuilder x)
{
x = null;
}

...

StringBuilder y = new StringBuilder();
y.Append ("hello");
Foo (ref y);
Console.WriteLine (y==null);

(Download sample code) Output:

True

Here, because a reference to y is passed rather than its value, changes to the value of parameter x are immediately reflected in y. In the above example, y ends up being null. Compare this with the result of the same code without the ref modifiers.

Now consider the struct code we had earlier, but using reference parameters:

void Foo (ref IntHolder x)
{
x.i=10;
}

...

IntHolder y = new IntHolder();
y.i=5;
Foo (ref y);
Console.WriteLine (y.i);

(Download sample code) Output:

10

The two variables are sharing a storage location, so changes to x are also visible through y, so y.i has the value 10 at the end of this code.

Sidenote: what is the difference between passing a value object by reference and a reference object by value?
You may have noticed that the last example, passing a struct by reference, had the same effect in this code as passing a class by value. This doesn't mean that they're the same thing, however. Consider the following code:

void Foo (??? IntHolder x)
{
x = new IntHolder();
}

...

IntHolder y = new IntHolder();
y.i=5;
Foo (??? y);

In the case where IntHolder is a struct (i.e. a value type) and the parameter is a reference parameter (i.e. replace ??? with ref above), y ends up being a new IntHolder value - i.e. y.i is 0. In the case where IntHolder is a class (i.e. a reference type) and the parameter is a value parameter (i.e. remove ??? above), the value of y isn't changed - it's a reference to the same object it was before the function member call. This difference is absolutely crucial to understanding parameter passing in C#, and is why I believe it is highly confusing to say that objects are passed by reference by default instead of the correct statement that object references are passed by value by default.

Output parameters
Like reference parameters, output parameters don't create a new storage location, but use the storage location of the variable specified on the invocation. Output parameters need the out modifier as part of both the declaration and the invocation - that means it's always clear when you're passing something as an output parameter.

Output parameters are very similar to reference parameters. The only differences are:

The variable specified on the invocation doesn't need to have been assigned a value before it is passed to the function member. If the function member completes normally, the variable is considered to be assigned afterwards (so you can then "read" it).
The parameter is considered initially unassigned (in other words, you must assign it a value before you can "read" it in the function member).
The parameter must be assigned a value before the function member completes normally.
Here is some example code showing this, with an int parameter (int is a value type, but if you understood reference parameters properly, you should be able to see what the behaviour for reference types is):

void Foo (out int x)
{
// Can't read x here - it's considered unassigned

// Assignment - this must happen before the method can complete normally
x = 10;

// The value of x can now be read:
int a = x;
}

...

// Declare a variable but don't assign a value to it
int y;

// Pass it in as an output parameter, even though its value is unassigned
Foo (out y);

// It's now assigned a value, so we can write it out:
Console.WriteLine (y);

(Download sample code) Output:

10

Parameter arrays
Parameter arrays allow a variable number of arguments to be passed into a function member. The definition of the parameter has to include the params modifier, but the use of the parameter has no such keyword. A parameter array has to come at the end of the list of parameters, and must be a single-dimensional array. When using the function member, any number of parameters (including none) may appear in the invocation, so long as the parameters are each compatible with the type of the parameter array. Alternatively, a single array may be passed, in which case the parameter acts just as a normal value parameter. For example:

void ShowNumbers (params int[] numbers)
{
foreach (int x in numbers)
{
Console.Write (x+" ");
}
Console.WriteLine();
}

...

int[] x = {1, 2, 3};
ShowNumbers (x);
ShowNumbers (4, 5);

(Download sample code) Output:

1 2 3
4 5

In the first invocation, the variable x is passed by value, as it's just an array. In the second invocation, a new array of ints is created containing the two values specified, and a reference to this array is passed.

Mini-glossary
Some informal definitions and summaries of terms:

Function member
A function member is a method, property, event, indexer, user-defined operator, instance constructor, static constructor, or destructor.
Output parameter
A parameter very similar to a reference parameter, but with different definite assignment rules.
Reference parameter (pass-by-reference semantics)
A parameter which shares the storage location of the variable used in the function member invocation. As they share the same storage location, they always have the same value (so changing the parameter value changes the invocation variable value).
Reference type
Type where the value of a variable/expression of that type is a reference to an object rather than the object itself.
Storage location
A portion of memory holding the value of a variable.
Value parameter (the default semantics, which are pass-by-value)
A value parameter that has its own storage location, and thus its own value. The initial value is the value of the expression used in the function member invocation.
Value type
Type where the value of a variable/expression of that type is the object data itself.
Variable
Name associated with a storage location and type. (Usually a single variable is associated with a storage location. The exceptions are for reference and output parameters.)

ASP.NET Validators Unclouded

2008-01-09T04:53:00.000-08:00

Introduction
It makes no difference what kind of development you are working on, user input validation is always an important and often troublesome issue.

What data needs to be validated? What are the valid conditions? Do we need to report every error in one pass or allow the user to correct one error at a time? Do we validate on the client or on the server? The questions go on.

Worse than that, the code can quickly get unmanageable, especially where we try to validate everything in one pass. Nested ifs, deconstructing strings, building up lists of error messages, endless comments trying to make some sense of it.

It is my personal opinion that everything that can be updated by anyone else but me should be validated at every opportunity and that we should always validate everything in one pass. But that does not mean that I have never tried to dodge the bullet on a tight deadline.

In some respects, it can be worse for the web page developer. It is important to validate as much as possible at the client to save on bandwidth, but HTTP requests are so easy to fake; the same data needs to be validated again on the server, if you want any kind of security.

Another problem can arise if the customer wants you to mark the invalid fields and then list the errors in one place (a common user-friendly convention in web-page design).

It is hard to imagine someone who enjoys writing validation code, and that goes double for classic ASP developers. Thankfully, the .NET framework has gone a long way to alleviate the problem for the ASP.NET developer with the introduction of six all-encompassing and developer-friendly validation controls:

RequiredFieldValidator Ensure that a mandatory field is populated.
CompareValidator Compare one control with another or with a literal.
RangeValidator Check that data is within a given range.
RegularExpressionValidator Use where data must match a given Regex format.
CustomValidator Use a custom validation function to validate a control.
ValidationSummary Collate the results of an entire form's validation.

This article assumes a basic knowledge of adding controls to a web form and setting the properties and event handlers. It attempts to give a general overview of these controls and how to use them to perform simple practical tasks.

All examples use C# with GridLayout pages. But anything in this article should easily translate to VB.NET and / or FlowLayout pages.

BaseValidator controls
All of the above, with the single exception of ValidationSummary, are derived from the BaseValidator class and share some functionality.

The first four validation controls are tailored for specific purposes and designed to validate an identical condition on both the client and the server. Each generates some purpose-specific JavaScript (using WebUIValidation.js, in the aspnet_client web) to validate the input control that you point to against the condition that the given validation control is designed to handle.

The CustomValidator allows you to define your own validation routines, using an event handler on the client, server or both.

The actual client-side validation takes place in events triggered by a change of value in the control you are validating and when you try to submit the form. The server-side validation takes place when you post back the form and the validation results can be accessed using the Form class's .IsValid property.

Each control's properties look reasonably similar. Apart from the standard font and formatting properties, each of the standard controls has the following common properties, derived from BaseValidator:

.ControlToValidate Name of the control we are validating.
.ErrorMessage The error message related to an invalid condition.
.Text Text to display in this control if invalid.
.EnableClientScript Validate client-side and server-side.
.IsValid The validation result of this control

ControlToValidate
A string containing the name of the Control object that the validation control will examine. This control must be developed using the ValidationPropertyAttribute.

Of the standard controls, the only ones marked with a ValidationProperty are:

TextBox
ListBox
DropDownList
RadioButtonList
HtmlInputText
HtmlInputFile
HtmlSelect
HtmlTextArea
It is worth bearing this in mind if you are ever developing a control that may require validation. You should always prefix the class definition with [ValidationProperty("MyProperty")] (where MyProperty is the name of the property that can be changed by the user input, usually text).

ErrorMessage
This defines the message related to the invalid condition. This can be displayed in the control itself or passed to a ValidationSummary control.

Text
The text to be displayed in the control when the condition is not valid. If left blank, the control will display the ErrorMessage text.

If the difference between .ErrorMessage and .Text is confusing, read on. It should make sense by the time we examine the ValidationSummary control. Until then, we will only use the .ErrorMessage property (.Text should be left blank)

EnableClientScript
By default this is set to true and the .ControlToValidate will be validated twice, at the client and at the server.

If the .EnableClientScript property is set to false, then validation will be limited to the server.

IsValid
The only property not available to the WebForm designer, .IsValid contains the validation result for this control after the form's validation has taken place.

This can be useful if you want to react differently when only part of the page is invalid.

Examples

The following sections will refer to a very simple example. As shown in the screenshot, this example will consist of a single TextBox being validated. You know the kind of thing: Leave your E-mail address here and we'll send you something really nice for free. (and then sell your email address to five hundred Email spammers)

Create a web form, containing only a Label ("Email Address:"), an empty TextBox, a submit Button and whichever -Validator control we are looking at. (see sample image)

The ControlToValidate property should always point to the TextBox and the ErrorMessage should contain something relevant to the validation we are working with.

Include some code in a Button_Click handler to redirect to another page when the button is clicked, but only if the page is valid. For example:

private void myButton_Click(object sender, System.EventArgs e)
{
if (this.IsValid)
Response.Redirect("anvokay.aspx", true);
}

Or... if you are feeling lazy, you can download the sample code at the top of the page and unzip it into a new or existing web.

NOTE: There is a strange behavior quirk in the interaction between web browsers and the .NET framework which means that if you hit Enter in a Single-TextBox form it will post back to the server but the Button_Click handler is not executed.

This is apparently a legacy browser issue where, for a single textbox form, the button name is not sent as part of the response, leaving .NET with no idea what triggered the postback.

Possible workarounds are:

Always physically click the Submit button rather than pressing Enter.
Include a second TextBox that does nothing, just to avoid this "feature".
Download and use MetaBuilders' Default Buttons Control to safely link the Submit Button to the TextBox's enter key.
(Thanks to Andy Smith for helping me understand this)

The RequiredFieldValidator
As the name suggests, the RequiredFieldValidator is normally used to validate that a mandatory field is populated. It can also be customized so that blank is a valid response and some other text means that nothing has been entered.

There is only one property unique to the RequiredFieldValidator:

.InitialValue Consider this text to be "no change" and blank to be "updated".

Example
Validate our simple form, using a RequiredFieldValidator, as seen in the example diagram above. If you hit the Submit button without entering any text, you can see that the form does not post back to the server, the JavaScript on the client-side catches the error and displays the error message that you gave it.

If you set .EnableClientScript to false and run the application again, then you will see that the submit button now forces a postback but the field is still validated, so if a malicious user attempts to forge the HTTP request to bypass your validation, it will not be successful.

And there you have it. You have now successfully implemented ASP.NET validation with one control and one line of code (if (this.IsValid)). How easy can it be?

InitialValue
The .InitialValue property gives you an opportunity to enter some text into the TextBox and still ensure that it is edited before submission.

Try changing both the RequiredFieldValidator.InitialValue and TextBox.Text to "EMAIL ADDRESS" and running the application again.

You should find that if you do not change this text and simply hit "Submit", you will get the error message as specified, but now if you blank the field then it is considered valid.

In many ways, this is a functionality overlap with the CompareValidator, as will become clear later on. But in some cases it is more sensible (i.e.. more readable) to use the RequiredFieldValidator - for example, validating that a DropDownList has changed from the "Please Select One..." default.

BaseCompareValidator controls
The CompareValidator and RangeValidator both compare values to literals or other user-entered data. The two controls inevitably contain some common functionality and thus are derived from the BaseCompareValidator class.

The BaseCompareValidator only offers us one property in addition to those offered by the BaseValidator class:

.Type Allows strings of specific formats to be compared accurately.

This one property, however, is very important and includes several "gotchas" that you might need to be aware of.

Type
The .Type property can be used to define how a string is treated. If a textbox is used for numeric, date or currency values, you can use one of the BaseCompareValidator controls to validate the type of data entered or compare it to a value of the same type more accurately.

e.g.. If compared directly as strings, "90" is greater than "100"; but if compared as numbers, "90" is clearly less than "100".

But be aware that both the Client and the Server validation will deal with the input data using the locale of the Server machine, unless you set the page or application Culture/UICulture properties (see globalization in MSDN).

If you do not set the Culture and, like me, you develop primarily on an English locale machine (date format: DD/MM/YYYY) and then run your sites live on a US locale server (date format: MM/DD/YYYY) then, as you can see from the screenshot above, the data will be validated differently in development from live.

The 10th of January is before the 1st of October, after all, whichever country you come from.

It is always worth noting on your page which format the user is expected to use (remembering that YYYY/MM/DD is considered culture-independent), but the more idiot-proof you make things, the bigger idiot the internet produces. You cannot guarantee that the user will pay attention to instructions and even if you ask for YYYY/MM/DD, you will still get someone trying to use MM/DD/YYYY without even suspecting that your machine has an English Locale.

It is strongly recommended that you always (even if you develop on the live machine, or one very similar) specify a Culture in the Web.Config file, for example:

requestEncoding="utf-8"
responseEncoding="utf-8"
culture="en-US"
uiCulture="en-US" />

Once you have done this, you know exactly where you stand. You can request the date in format MM/DD/YYYY format and if the user chooses to ignore that and use a culture-independent style then there is no harm done.

If you do not specify a culture, even in the @Page clause, then you need to be especially careful when comparing to a literal, which may be treated differently in development from production (in fact may cause an exception, if the locale is different and the value becomes invalid). You must then use a culture-independent style for literal dates.

All of this applies equally to numerics in countries which take a comma (,) as a decimal point and to systems using different currency symbols.

Another thing to watch out for is that if any of the values being compared are not of the type the validation is expecting (and in the case of dates, this includes DD-MMM-YYYY, eg. 10-Jan-2003), the form data will be considered valid. The reason for this is that an extra CompareValidator (using the DataTypeCheck operator described below) gives you a way out of this and you would generally expect the error message to be different.

The CompareValidator
The CompareValidator is, at first glance, the least useful of the standard validation controls. However it does have its place and has plenty of designer properties (including the BaseCompareValidator.Type property, see above) to tweak its functionality in interesting ways.

.ControlToCompare Compare the value in this control to the value of another control.
.Operator In what way should we compare the values?
.ValueToCompare Compare the value with a literal value.

Example
In the simple example, we can use this control to ensure that no one uses a specific E-mail address in the form (I mean, we don't want someone registering us for our own spam E-mail scam, do we?).

Drop a CompareValidator in place of the RequiredFieldValidator; set the .Operator to NotEqual (why Equal would ever be the default is beyond me!) and the .ValueToCompare to your own E-mail address. Do not forget, of course, to set the .ControlToValidate and .ErrorMessage properties, as before.

When you run the application and try to enter your own email address into the form, the validator should give you a message as soon as you leave the TextBox. It should also refuse to let you submit the form to the server.

As pointed out before, this is not entirely useful in itself; for the rare occasions you would use this, you would not mind hacking the RequiredFieldValidator to do your bidding.

Only when you start considering the combinations of the unique properties listed above do you realize how powerful this is.

ControlToCompare
Instead of comparing a control with a literal value, you can compare it to another user-entered string. You do this by pointing the .ControlToCompare property to the control you wish to validate.

If this and .ValueToCompare are both set, .ValueToCompare will be ignored.

Operator
It is not only possible to compare values for equality (or inequality), it is possible to look for GreaterThan, GreatThanEqual, LessThan or LessThanEqual.

In all cases, if the condition is true then the .ControlToValidate is considered valid.

Alternatively, you can even check that the input data can be converted to a certain type, using the DataTypeCheck value in this property. This is essential, as pointed out above, because a comparison including an invalid value will always be treated as valid.

For example, when comparing two dates, if "F" is compared to "01/01/2003" then it is simultaneously considered to be greater than, less than and equal to. But when checking "F" using a CompareValidator where .Operator = DataTypeCheck and .Type = Date, validation will fail.

When using .Operator = DataTypeCheck, the .ValueToCompare and .ControlToCompare properties are both ignored and the .Type property is used to define the valid type.

The RangeValidator
The RangeValidator is much like the CompareValidator control (hence the common base class), except that it verifies that the user data falls between two constant values, rather than comparing it to a single value (constant or variable).

With that in mind, the control's unique properties (not part of BaseCompareValidator) are much as you might expect.

.MaximumValue Value may not be greater than this.
.MinimumValue Value may not be less than this.

Example
Referring to our simple form, we can easily use a RangeValidator to verify that the email address entered begins with a lower-case alpha character. Of course, this is not much validation for an email address, but it is a good demonstration of the RangeValidator.

Replace whichever validator you are using with a RangeValidator, set the .ControlToValidate and .ErrorMessage as for any validator. Then set .MinimumValue to a and .MaximumValue to z.

Now any string entered that begins with a lower-case alpha character will be considered valid. Note again that if the TextBox is left blank, it is still considered valid. A separate RequiredFieldValidator must be used if the field is mandatory.

MaximumValue
The value entered by the user may not be greater than this, but the two values can be equal.

MinimumValue
The value entered by the user may not be less than this value (unless blank), but the two values can be equal.

The RegularExpressionValidator
Arguably the biggest, most versatile weapon in the ASP.NET validation arsenal has to be the RegularExpressionValidator.

With just one custom property over those provided by the BaseValidator class, a world of options become available.

.ValidationExpression Value must be in the format described.

Example
In our simple example, replace the current validator with a RegularExpressionValidator and set the .ControlToValidate and .ErrorMessage as usual.

Set the .ValidationExpression to \w+([-+.]\w+)*@\w+([-.]\w+)*\.\w+([-.]\w+)*. If you are using Visual Studio .NET, you can click the ellipses (...) button alongside the property and select Internet Email Address from the default list.

Whilst not perfect email address validation (see below), this is a pretty good approximation. It ensures that the address consists only of "word" characters (A-Z,a-z,0-9,_) with exactly one @ symbol and at least one period (.) after it, allowing for variations which include hyphenations in mid-word and extra periods before and after the @ symbol.

Again, notice that a blank field is always valid unless you also use a RequiredFieldValidator. And, as with all validation controls, no code is required to perform an identical validation on both the Client and the Server.

ValidationExpression
The data entered, if not blank, must match the format specified in regular expression format.

This article will not attempt to cover regular expressions in detail, much has been written elsewhere on this site and others on this subject.

Frankly, I do not claim to understand it that well, I usually manage with the default values provided by Visual Studio .NET or others that I find through Google (there are some fascinating expressions out there, if you look hard enough).

The Visual Studio .NET default values can be found via the ellipses (...) button to the right of the property (within the property sheet) and offer the following validations:

US / French / German / Japanese / PRC Phone Number
US / French / German / Japanese / PRC French Postal Code
US / PRC Phone Number
Internet Email Address
Internet URL
There are some surprising omissions here (certain nationalities, US State Abbreviations, ISBN, Credit Cards, Long Date Formats), and some of the default expressions (inc. Email address) are not 100% accurate.

The Regular Expression Library is the best resource I have found for most of those that are missing and for improving those that exist.

If anyone has other resources, I would be happy to list them with the next update.

The CustomValidator
Sooner or later (probably later, to be honest), you are going to need to validate something and the standard validators are not going to offer it. Maybe you need to access a database, or perform some complex validation involving numerous controls.

If you are going to need to do it repeatedly, you might want to consider creating your own validation control, but for the one-off oddity you can use the CustomValidator.

The CustomValidator allows you to add code to be run on the client and / or the server and set a value showing the result.

You can populate the .ControlToValidate property, passing the value of the control to the event handler or script. But, unlike the other validation controls, the CustomValidator will not throw an exception if .ControlToValidate is empty.

Other than that, it acts exactly the same as any other validation control, either showing the .ErrorMessage in the control itself or passing it to a ValidationSummary control.

The CustomValidator has only one unique property and a unique event handler.

.ClientValidationFunction Name of HTML-scripted function to execute, on validation
.ServerValidate Event handler for server-side validation.

Example
Using our simple example once more, replace the validation control with a CustomValidator. Set the .ControlToValidate and .ErrorMessage properties as before. Ignore the .ClientValidationFunction property for now and add an event handler for the .ServerValidate event.

You can validate the value entered against a database, if you wish. For simplicity, I have chosen to simulate this with a simple if statement.

private void alreadyInUse_ServerValidate(object source,
System.Web.UI.WebControls.ServerValidateEventArgs args)
{
// Default Value
args.IsValid = true;

// Simulating a check against a database
if (args.Value == "pdriley@santt.com")
args.IsValid = false;
}

The validator will always post back to the server to the ClickEvent handler which the requests the results of the validation. But if you enter one of the "existing" E-mail addresses (in my case, my own address) then the error message will be reported back and the Response will not be redirected.

NOTE: The ServerValidate event itself, in fact all server-side validation, is executed after the Page_Load event and before any other events. But the result of the validation is irrelevant unless you later check that the form is valid. The Button_Click event, in this case, is still executed.

ServerValidate (Event)
As you can see in the example above, the ServerValidate event handler should accept an object (the CustomValidator) and arguments of type ServerValidateEventArgs.

The ServerValidateEventArgs class exposes two properties: .IsValid and .Value. All you have to do in most cases is check that .Value is valid and set .IsValid accordingly.

If there is no .ControlToValidate property set for the CustomControl then args.Value will default to String.Empty, but you can still validate the values contained in the controls.

ClientValidationScript
Writing a JavaScript handler is conveniently similar to writing the server-side handler. It still exposes the same two arguments, source and args.

The sample C# code above would directly translate as:

function alreadyInUse_Validate (source, args)
{
// Default Value
args.IsValid = true;

// Simulating a check against a database
if (args.Value == "pdriley@santt.com")
args.IsValid = false;
}

All that is left to be done is set the .ClientValidationScript property to alreadyInUse_Validate and the client-side validation is complete.

The ValidationSummary
It should be fairly clear by now that many web pages are going to include a lot of validation controls. No user is going to be happy with all those error messages popping up at random positions around the page, but it is always nice to have something marking the invalid fields.

And this is where the ValidationSummary control comes into play.

The ValidationSummary derives directly from WebControl and is not related to the other controls except by symbiotic functionality.

When a page is validated, all of the .ErrorMessage properties from validation controls in the same container (e.g.. Form or DataGrid row) that fail validation are passed to the ValidationSummary control to be displayed in an unordered list.

This finally explains the .Text property, common across the validation controls. If .Text is populated then it will be displayed in place of the validation control, but still .ErrorMessage will be used to populate the ValidationSummary.

The ValidationSummary control has a number of unique properties that affect the way messages are displayed to the user.

.DisplayMode How the unordered list is rendered.
.EnableClientScript Should the error list be updated during client validation?
.HeaderText Defines the text that heads the error list.
.ShowMessageBox Show error message list in a javascript "alert".
.ShowSummary Render the error messages on the page itself.

Example
To demonstrate the full power of the ValidationSummary, it is necessary to come up with something a little more complicated than the simple example that is used throughout the rest of the article.

Consider a simple registration form for, say, a chat room.

The customer requesting this form has asked for the following:

User ID - mandatory, must be 6-8 characters
Password - mandatory, must be 8 characters with at least two numeric, also confirmed
Name - mandatory
E-mail - mandatory, must be valid format
Sex - mandatory, drop down list, must not default
Date of birth - not mandatory but must be valid if entered

Set the form up yourself, complete with validators, using the rest of this article as a guide. As well as giving each control an error message, change the .Text properties to * and drop the new (much smaller) control next to the control it is validating.

Although the Visual Studio .NET IDE will not allow you to drop validation controls directly on top of each other, you can edit the HTML view to force the controls into the same position.

Along with the diagram, here is a clue: there are 12 validation controls in total:

6 x RequiredFieldValidator
4 x RegularExpressionValidator
2 x CompareValidator
Drop a ValidationSummary at the bottom of the page, next to a submit button. The only property you might want to set is the .HeaderText. (In the example, it is set to Please fix the following errors:)

Run the form and you can now see how a combination of validation controls and a ValidationSummary control can be used to create a very secure and yet flexible and most importantly usable Web Form.

If you get stuck then, again, the example is included in the downloadable zip file at the top of the article.

DisplayMode
The list can be rendered in any of the following DisplayMode styles:

BulletList (list items in an unordered list)
List (delimited only by a new line)
SingleParagraph (delimited by space)
The default option is BulletList.

EnableClientScript
By default, .EnableClientScript is true and the list of errors will be shown when the form is submitted.

If .EnableClientScript is set to false then the ValidationSummary will not be shown until the server reports the errors back to the client machine.

This is useful if (and only if) the .EnableClientScript property of all the validation controls are also false. If they are not then the form will not post back to the server and the ValidationSummary will never report the errors.

HeaderText
Allows the definition of text to describe the list of errors. This will be posted directly before the list of error messages, regardless of the .DisplayStyle value.

ShowMessageBox
If set to true, the ShowMessageBox property creates a JavaScript, alert() call to display all of the error messages in a message box.

The message box will emulate the style defined in .DisplayMode as closely as possible.

ShowSummary
If set to true, the ShowSummary property renders the list of error messages into the HTML at the position of the Label.

Either ShowMessageBox or ShowSummary should always be true, but they should only both be true if absolutely necessary.

Bypassing validation
If you have two postback events (e.g.. "Next" and "Back" buttons), and one requires validation while the other does not, you can set the .CausesValidation property of one control to true and the other to false.

This will bypass both client and server validation. You can still check this.IsValid, if you want to use some common code for handling both events, and validation will succeed even if there is no valid data entered.

Conclusion
This is a very powerful set of controls with very few obvious dangers. As an added bonus, they are incredibly easy to implement. In Visual Studio .NET, it is just drag-drop and select a couple of properties.

There is a lot of information to absorb in one pass, but hopefully this article will serve as a reasonably simple reference guide as well as a "How to" guide.

Once you have used the validation controls a few times, you will be dropping them into pages just for fun.

Downloads

Safely Storing Business Objects in Session State

2008-01-08T01:22:00.001-08:00

Introduction
This article describes a method for creating robust ASP.NET applications when business data is stored in Session state and users may open multiple browser instances from the same desktop client. Without protection, business data may get out of sync with a particular browser window if it is altered through another browser window. The solution presented in this article supplements the ASP.NET Session ID with a unique sequence ID string written out to each page on every post-back just before it is rendered to the client browser. Different instances of business data stored in Session state are indexed by this sequence ID string and are mapped to the correct browser windows using the sequence ID found on the page. The solution presented here is stable against multiple instances of web forms opened up by users on different instances of different business data, different instances of (possibly) the same business data, or popping cloned windows with CTRL-N.

Background
Some time ago, I developed a web form for internal use at our institution to help staff members gather complex service data from customers. The application architecture is such that the web form is driven by a controller which keeps it in sync with its stored in Session state. Within a month of rollout, one of the users tried keeping multiple browsers open on at least two distinct instances of business data, in order to copy and paste from one to the other. Since there was only one instance of business data in Session state at a time, this resulted in data corruption.

ASP.NET can keep application data in server memory based on a user's Session as identified by a long string called a Session ID. The Session ID is often stored in a cookie, but it is also possible to store the session in a URL string. You can see the Session ID in Visual Studio by examining the Page.Session.SessionID string. The above problem occurs when multiple browser windows having the same session ID are open at the same time within the same web application. It can happen with or without cookies, although it happens more transparently with cookies because you don't ordinarily see the value of the Session ID. Also, cookies are indexed on disk by application URL and are thus transparently shared by multiple browsers running in the same web application.

It is instructive to think about the web application as a sequence of discrete user operations running against distinguishable instances of business data. In order to prevent user operations from being applied to the wrong instance of business data, all user operations must be identified with a particular sequence, and applied only to the instance of business data corresponding to that sequence. In this respect, it is interesting to compare the ASP.NET web application to a traditional machine-bound OO program. In both the ASP.NET web application and the OO program, there is logical data encapsulation. Based on the types of objects involved, methods will happily run against object data if the types and method signatures agree. There is only limited physical data encapsulation in the web application, however. In the traditional OO model, methods are run against object data based on object identity kept in pointer tables maintained by an interpreter or produced by a compiler. In an ASP.NET web application with state, we have only one Session ID and it's not fine enough on its own to distinguish among multiple instances of business data in the same Session.

The solution is to supplement the ordinary ASP.NET Session ID with a secondary sequence ID string. This allows the identification of a sequence of user operations and maps it to one of several stored instances of business data in Session state. Since a user generally thinks of operations within an individual browser window as a distinct set, we'll write this sequence ID out to the browser window using ViewState in an ASP.NET HiddenField control. The sequence of logical user operations can thus be identified with a sequence of post-backs.

The code behind base class
The above solution is most conveniently implemented as a base class to your code behind pages. The machinery behind it applies to any web page.

public abstract class CodeBehindBase : System.Web.UI.Page
{
private string sequenceID;
public string SequenceID
{
get { return sequenceID; }
}
private string NewSequenceID()
{
// Just using the clock with one second resolution is
// sufficient for most cases.
// If you need to worry about more that one browser
// window open per second,
// use milliseconds.
return DateTime.Now.ToString();
}
// (...)
}

The class defines the sequence ID and a private method for creating new sequence IDs. Although it may seem that some sort of business moniker may be needed to tie a sequence of user operations to an instance of business data, it is not the case. All that is needed is a relatively unique identifier that can be tied to a particular sequence of post-backs. The mechanism uses the same sequence ID to index instances of business data in memory, so there is no dependence of this solution on how objects are named in the business model. The reason that the base class is abstract will be explained below.

The code behind base class emits an ASP.NET HiddenField control to the control tree during OnInit. This field will be used to write the sequence ID out to the browser. A dictionary to be used to index different instances of business data is also initialized and stored in Session state.

protected sealed override void OnInit(EventArgs e)
{
// Adds a hidden field to the control tree to hold the
// sequencce number
// in ViewState so we can identify the browser window.
// Must be added here
// or else it will miss having ViewState loaded.
HiddenField hdSequenceID = new HiddenField();
hdSequenceID.ID = "hdSequenceID";
hdSequenceID.Value = "";
Form.Controls.Add(hdSequenceID);

// Initializes the data structure for linking sequence IDs to
// form data
if (Session["FormDataObjects"] == null)
{
Session["FormDataObjects"] = new Dictionary();
}

// Page_Init is called here if it is defined in the derived class.
base.OnInit(e);
}

Note that this amounts to storing the Session ID in the ViewState of a dynamically created control. The control must be created and added to the control tree during OnInit so that ViewState is available in OnLoad on subsequent post-backs. Instances of the business data must be indexed by the same sequence ID, so the code behind also has rudimentary facilities for handling business data. To avoid dependencies on the business model, this is done at the level of object.

protected abstract object NewFormData();

private object GetFormData(string sid)
{
Dictionary formDataObjects =
(Dictionary)Session["FormDataObjects"];
if (formDataObjects.ContainsKey(sid))
{
return formDataObjects[sid];
}
return null;
}

protected object GetFormData()
{
object retval = GetFormData(sequenceID);
if (retval == null)
{
throw new Exception(
"Form data was not found in the session." +
"This could have been caused by " +
"opening a duplicate window using \"Ctrl-N\"." +
"Close this window and the duplicate window " +
"should be OK.");
}
return retval;
}

Derived classes must implement the NewFormData() method. This method will be called when the page is accessed for the first time to load blank form data. Since the details of object creation may be business model dependent, this method is left abstract so the developer of a derived class must implement it. In turn, when accessing business data, derived classes must use the GetFormData() method to get the instance corresponding to the value of the Session ID. Note that the only two "special" methods the derived class developer needs to know about are NewFormData() and GetFormData(). If a particular business object was not found in session state, it is usually because its corresponding sequence ID has been removed or updated. This most often happens when the user opens a window clone using CTRL-N, works with the clone for a while, and returns to the original window. Working with the clone will have caused the sequence ID to be updated and the original window will become invalid.

On page load, ViewState will be available so that the sequence ID can be read from the post-back. The sequence ID is set directly on the Page object. If the page is not a post-back, then the sequence ID just gets whatever value was set during OnInit (usually the empty string.) When the page is not a post-back, further processing ensures that a new instance of business data is loaded and a new sequence ID is generated for it. It uses two routines, LinkFormData() and UnlinkFormData(), to maintain the correspondence between instances of business data and sequence ID stored on Page.

Collapse protected sealed override void OnLoad(EventArgs e)
{
// Set the SequenceID coming back from the page.
sequenceID = ((HiddenField)Form.FindControl("hdSequenceID")).Value;

if (!Page.Ispost-back)
{
// Get an instance of form data to use
object formData = null;
if (Page.Request.Params["SequenceID"] != null)
{
// If this is a Server.Transfer, check if the SequenceID is
// in the query string.
// If so, load the form data from the old page.
// If you're worried about spoofing,
// you'd probably want to turn this off.
string sid = Page.Request.Params["SequenceID"];
formData = GetFormData(sid);
UnlinkFormData(sid);
}
else
{
// Otherwise get new form data.
formData = NewFormData();
}

// If not a post-back, then the seuqenceID would have the
// value set in OnInit.
// So create a new sequence ID here and link it to the form data.
sequenceID = NewSequenceID();
LinkFormData(sequenceID, formData);
}
// Calls Page_Load()
base.OnLoad(e);
}

The code also contains sugar to allow for instances of business data to be maintained across a Server.Transfer(). In that case, make sure the transferring page references the SequenceID in the URL query string. At the PreRender stage, the code ensures that a fresh sequence ID is written out. This is to prevent problems with the user obtaining browser clones using CTRL-N. Since clones copy ViewState, clones get the same sequence ID. Unless the sequence ID changes on every post-back, business data can once again change underneath windows copied with CTRL-N.

protected sealed override void OnPreRender(EventArgs e)
{
// Regenerate the sequenceID on every post-back.
// This prevents problems arising when the
// user opens up multiple browser windows with "Ctrl-N".
// The new window will still work,
// but older windows will safely error out.

// Get the form data
object formData = GetFormData(sequenceID);
// Unlink the old sequence ID
UnlinkFormData(sequenceID);
// Generate a new sequence ID
sequenceID = NewSequenceID();
// Relink the form data
LinkFormData(sequenceID, formData);
// Write the sequence ID out to the page.
((HiddenField)this.FindControl("hdSequenceID")).Value = sequenceID;

// Calls Page_PreRender.
base.OnPreRender(e);
}

The LinkFormData() method mentioned above will use the object Equals() method to determine when an attempt is made to link multiple instances of the same business data when the business layer thinks they are the same. Using application means, a user may for example open multiple instances of the same customer record. If it is important to try and prevent this occurrence, the developer can override Equals() on the business data object.

private void LinkFormData(string sid, object formData)
{
Dictionary formDataObjects = (Dictionary object>)Session["FormDataObjects"];
foreach (object o in formDataObjects.Values)
{
if (formData.Equals(o))
{
throw new Exception(
"Tried to load multiple instances of the same form data.");
}
}
formDataObjects[sid] = formData;
}

Using the code
The demo project associated with this article requires Visual Studio 2005 and .NET 2.0. It contains the above code behind base class and two test pages that can be opened and run within Visual Studio. The test application is basically just a text box with an "Append" button that allows users to append characters to a string. The business data class is simply an instance of System.Text.StringBuilder. Note that the string class is often a bad choice for these kinds of demos because it is a read-only sealed reference type. Appending to a string therefore necessarily involves the implicit creation of new strings and often makes things not so clear for a demonstration.

The first test page, BadSession.aspx, does not use the code behind class. To see the Session state problem, open BadSession.aspx in a browser window and type in some data. To open up a second window, use the "Pop" button. Work with the second window for a little while and go back to the first window. Data from the second window will have contaminated the first window through the common shared Session state. To see the problem again in a slightly different way, use CTRL-N instead of "Pop." The only difference is that "Pop" will additionally clear business data, since the initial pop is not a post-back.

The second test page uses the code behind base class developed above. The only additional difference is that instead of loading a new instance of StringBuilder into Session state during PageLoad, it overrides the NewFormData() method and lets the base class do it. Subsequently, when it needs the business data, it uses GetFormData() instead of accessing Session state directly. Each new window (using "Pop") now corresponds to a distinct instance of form data. If the user uses CTRL-N to clone a window, then the first clone to generate a post-back grabs the sequence ID and the rest of the clones become safely invalid. This is why the sequence ID is regenerated on every post-back.

The reader is invited to try this out with a custom business class with overridden Equals(). Care must be taken how to define equality here. For example, if equality is determined only by the content of business data, then there is a danger that separate evolutions of business data may wind up having similar enough content to be considered the same when really they're not. It is best to define equality based on some known unique identifier, like a database primary key.

Points of interest
The implementation above uses ViewState to store the sequence ID. If the developer is worried about ViewState being turned off, then the implementation can be changed to use ControlState in the Hidden field control.
The implementation uses a date string as the sequence ID. Any sequence ID will do, as long as it is distinct on every post-back for all open windows. It does not have to be a globally unique ID, just unique within the scope of the web application running client-side.
The developer may elect to override Equals() in the business data object. This can prevent multiple instances of the same business data -- as defined by business rules -- from being linked to multiple windows at the same client. It does not prevent multiple windows from being open on the same business data on different clients, since they will have different Session IDs to begin with. It is beyond the scope of this article to address that.
The "sequence ID" itself doesn't have to be a constant identifier or even part of a sequence. The user operations are the sequence that we're interested in and the "sequence ID" merely denotes the fact that the user operations need to be mapped to a particular instance of business data.
Old instances of business data are not garbage collected from Session state in this implementation. If the sequence ID implementation uses a timestamp, a scan can be be added to OnLoad to remove old business data instances when their age exceeds some timeout period.

A Session Data Management Tool

2008-01-08T01:21:00.001-08:00

Introduction
In an ASP.NET application, it is very easy to pass session data from one web page to another. For example, the following code stores three pieces of data of an internet session using the .NET web session provided by the .NET framework:

...
Session("UserName") = sUserName
Session("PageName") = sPageName
Session("StatusCode") = nStatusCode

If later the user navigated to a different page, there is no need to pass the session data via the query string because they are already available on the server. The following code will retrieve the session data stored previously:

...
sUserName = CType(Session("UserName"), String)
sPreviousPageName = CType(Session("PageName"), String)
nPreviousStatusCode = CType(Session("StatusCode"), Integer)

By default, the session data is stored in memory of the server process, which makes it very efficient. If you want load balancing your applications using several servers or you want to be able to reboot the server machine quickly without big impact on current users, you can modify the web.config file of your application so that the session data will be stored in a SQL server database or in the memory of a Microsoft session state server instance.

It seems that we always want to do more than the current tool allows. For example, I would like to share session data among different applications. Some of my applications are running on machines without the .NET framework and some of them are not even web applications (windows services, console apps, and desktop GUI apps, etc.). Note that we are only dealing with applications we trust so there is no security issue here.

The .NET framework built-in session is not suitable for me because

It does not allow you to share session data among different applications. If you use the SQL server option, then it is possible to hack the stored procedures provided by Microsoft to allow session data sharing (but this is not an approach I like).
It is only available to .NET web applications, other applications cannot use it.
Using the SQL server option, session data is saved into a database. However there is no way for you to view the data outside of your application, because it is stored in binary format and you don't have access to Microsoft's source code.
In this article, I will introduce a simple, reliable, and efficient session tool that does not have these shortcomings. I could have any number of applications running on one or more servers all sharing the same session data. I hope this tool will be useful to you, too.

Overview of my session tool
The session data is managed by a .NET library Session.dll which contains the SessionManager class. The capability of sharing session data among applications is provided by a .NET web service called SessionService which uses the Session.dll component internally. This web service can also function as a .NET remoting server.

All session data is stored in the memory of the SessionService process. Theoretically the performance of the SessionService should be between the .NET in-memory session and the .NET SQL server session. However, the SessionService also saves session data into a local file periodically to prevent data loss due to server crash or reboot, etc.

Different applications can store and retrieve data from this SessionService. .NET applications can take advantages of the remoting capability to use more custom data types. Session data used by client programs won't be lost even if you reboot the server machine. Expired sessions will be cleaned up automatically. It is possible to set the timeout value for each individual session.

I have included with this article some sample code to access the SessionService. There is also a SessionBrowser program which can be used to view and change session data stored in the SessionService .

The SessionManager class
This is a class implemented in Session.dll written in VB.NET (actually it can be converted easily to any other .NET language in case VB is not your favorite choice). It implements the SetSessionData and GetSessionData methods

Public Function SetSessionData( _
ByVal sID As String, ByVal sKey As String, _
ByVal oValue As Object) As Boolean
Public Function GetSessionData( ByVal sID As String, _
ByVal sKey As String) As Object

The C# signatures are

public bool SetSessionData(String sID, _
String sKey, Object oValue);
Public Object GetSessionData(String sID, String sKey);

The sID parameter is a string that uniquely identifies a session, a guid string is a perfect candidate to be used as a session id. The sKey parameter must be a unique string within the corresponding session. As you can see, these two methods work with general .NET data objects (not just strings and integers). To store and retrieve session data, you do the following:

...
SetSessionData(sMySessionID, "UserName", sUserName)
SetSessionData(sMySessionID, "PageName", sPageName)
SetSessionData(sMySessionID, "StatusCode", nStatusCode)

...
sUserName = CType(GetSessionData(sMySessionID, _
"UserName"), String)
sPreviousPageName = CType(GetSessionData(sMySessionID, _
"PageName"), String)
nPreviousStatusCode = CType(GetSessionData(sMySessionID, _
"StatusCode"), Integer)

Here is the same code in C#

...
SetSessionData(sMySessionID, "UserName", sUserName);
SetSessionData(sMySessionID, "PageName", sPageName);
SetSessionData(sMySessionID, "StatusCode", nStatusCode);

...
sUserName = (String)GetSessionData(sMySessionID, "UserName");
sPreviousPageName = (String)GetSessionData(sMySessionID, "PageName");
nPreviousStatusCode = (int)GetSessionData(sMySessionID, "StatusCode");

Internally, the SessionManager class stores sessions in a shared (static for C#) hashtable. Each entry in the hashtable is itself a hashtable which contains data objects for a particular session. There is no need to create a session, it will be created when SetSessionData is called if the session does not exist already. The GetSessionData method will return Nothing (which is the same as null in C#) when the requested data does not exist.

Each session has a time stamp associated with it. When the SetSessionData and GetSessionData methods are called, the corresponding time stamp is updated. The session will expire if it has not been accessed for a specified amount of time. The SetTimeout method determines the expiration interval, the default is 30 minutes. When you call GetSessionData for an expired session, it will return Nothing as if the session was never created. The SetSessionTimeout method can be used to set the timeout interval for an individual session. The individual timeout interval always overrides the global timeout interval. Therefore you could have one session that expires in 10 years while the rest will expire in 30 minutes. The RemoveSession method is used to remove a session with given id.

The SaveSessions and LoadSessions methods are used to write all sessions to a file and read them from a file, they take one parameter that is the full path of a session data file . The SetSessionDataFile sets the path of the internal session data file. If you call the method OnStart at the beginning of your application, it will read session data from this internal session data file. The OnStart method will also launch a background thread that removes expired sessions to keep the memory usage from getting out of control. The OnStop method will save all current sessions to the internal session data file and kill the background thread. So if you put the calls to the OnStart and OnStop methods at the appropriate places in your code, you can shutdown and restart your app without losing any session data.

Note that if you want to persist session data to a file or you want to store and retrieve session data via the SessionService, all your session data objects have to be serializable.
If you have a serializable object implemented in your own class and you want to store it as session data using the SessionService, the dll that implements this class has to be copied into the Bin folder of the SessionService, otherwise serialization won't work properly.
All .NET applications, which do not have to be internet related, can use Session.dll within the same process as the application. The SessionManager class is derived from MarshalByRefObject, which means you can set up a .NET remoting server and access session data from various applications. Applications using the same .NET remoting server can access all session data hosted on that server, as long as the correct session id and session key are supplied when calling GetSessionData and SetSessionData .

To set up a remoting client, you need to create a configuration file containing the following text:

url=" http://localhost:80/SessionService/SessionManager.soap" />

The url parameter determines where the remoting server is located. At the beginning of your application, you need to have the following code

RemotingConfiguration.Configure(sConfigurationFilePath)

where sConfigurationFilePath is a string representing the full path of the configuration file. Then you can simply use the New method to instantiate a SessionManager object and call the SetSessionData and GetSessionData methods

...
Dim mgr As SessionManager = New SessionManager( )
mgr.SetSessionData(sSessionID, "TestData", "This is a test")
sData = CType(mgr.GetSessionData(sSessionID, "TestData"), String)

Here is the C# version

... SessionManager mgr = new
SessionManager(); mgr.SetSessionData(sSessionID, "TestData", "This is a
test"); sData = (String)mgr.GetSessionData(sSessionID, "TestData");

Note that once you configured the remoting client, session data will be saved to and loaded from the remoting server. Using the remoting server is not as efficient as keeping session data in memory of the same process, but it allows multiple .NET applications to share the same data. Also, using a .NET remoting server limits your session objects to serializable objects. We will talk about configuring the remoting server later. The disadvantage of using a .NET remoting server (comparing to using the web service) is that the clients can only be .NET applications.

Since multiple applications can access the same data via SessionManager , the actual data doesn't have to belong to any real user session. In fact, we can use this tool to read and write global data as demonstrated by the following code

...
Dim mgr As
SessionManager = New SessionManager( )mgr.SetSessionData("GlobalData", "RootDir", "c:\AllOfMyApps")
Dim sConn as String = CStr(mgr.GetSessionData("GlobalData", "ConnectString"))

Here is the C# version

...SessionManager mgr ;= new
SessionManager(); mgr.SetSessionData("GlobalData", "RootDir", "c:\AllOfMyApps");String sConn = (String)mgr.GetSessionData("GlobalData", "ConnectString");

The web service and the remoting server
The SessionService component provides the main features of the SessionManager class through a .NET web service. With this web service, you can have applications running on non .NET machines all sharing the same set of session data. You can even have applications accessing session data across firewalls.
To install the web service, you need to do the following

Unzip the files to the server machine. It must have the .NET framework and IIS installed.
Add a virtual web directory pointing to the SessionService subdirectory.
That's it. Now you can call the web service from client programs. Another way to install SessionService is run the included Install.vbs script.

The SessionService will call the OnStart method of SessionManager automatically when it is first accessed. The OnStop method of SessionManager will be called when the web service is shutdown. Therefore restarting IIS or even rebooting the machine will not cause any session data to be lost, unless there is an abnormal termination of IIS, of course.

For non .NET client programs, calling the SessionService is made easier by using XYSoapClient.dll. This is a regular com dll that can be used in C++, VB, and VB script to access almost any web service. For example, here is a script that set and get session data:

Dim obj
Set obj = CreateObject("XYSoapClient.1")
obj.InitService "http://localhost:80/SessionService/SessionService.asmx?wsdl"
If obj.InvokeMethod("SetSessionString", "MyID", "MyKey", "Test") Then
wscript.echo "Set session data ... Click OK to verify the data"
wscript.echo obj.InvokeMethod("GetSessionString", "MyID", "MyKey")
End If
Set obj = Nothing

The XYSoapClient.dll is described in detail in another article.

You can have multiple instances of the SessionService running on different machines or on the same machine. A limitation of the SessionService is that you can only use string data with non .NET clients. This is not a big deal because you can always represent other data types using xml, it is not possible to use .NET objects in a non .NET program anyway.

For .NET client programs. You can add web reference to access the SessionService. Alternatively, you can use the SessionProxy class. Both VB.NET and C# versions of this class are included with this article. Here is the VB.NET sample code

Dim proxy As SessionProxy
= New SessionProxy("http://localhost/SessionService/SessionService.asmx")
proxy.SetSessionData("MyID", "MyKey", "Test")
Dim sTestData As String = CStr(proxy.GetSessionData("MyID", "MyKey"))

And here is the C# version

SessionProxy proxy = new SessionProxy(
"http://localhost/SessionService/SessionService.asmx");
proxy.SetSessionData("MyID", "MyKey", "Test");
String sTestData = (String)proxy.GetSessionData("MyID", "MyKey");

If you look carefully, you will find the following text in the web.config file of SessionService:

type="SessionLib.SessionManager, Session"
objectUri="SessionManager.soap" />

The above text exposes the SessionManager object to .NET remoting clients. In other words, this web service is also a remoting server. Once you installed SessionService , you can have non .NET clients calling the web service while at the same time have .NET clients create SessionManager objects remotely. There is no need to set up a separate remoting server. Isn't that nice?

If you unzip the downloaded file to a folder and run the Install.vbs file, it will copy all the components to a folder of your choice and set up the web service and sample applications for you (creates virtual directories, etc.).

Error logging in the session service
The previous version of the SessionService writes error messages into the event viewer. The current version will create trace files in the Log folder. Information written into the trace file can be errors (level = 10), warnings (level = 20), or debug info (level = 30 or 40). You can configure the tracing capability of the SessionService from the web.config file.

Here is the section in the web.config file that determines how tracing is done.

The server will generate a new trace file each day. The TraceFilePrefix parameter determines where the trace files will be generated. The TraceLevel parameter determines how much information will be written to the trace file, the typical values are 0, 10, 20, 30, and 40 (0 means no tracing, 40 means the most tracing). The TraceCleanup parameter determines how long the trace files will be kept, the value 7 in the above means all trace files will be deleted automatically after 7 days.

Test programs and the SessionBrowser
The program SessionRemotingTest.exe and SessionConsoleTest.exe demonstrate how to access the SessionService via remoting from web and console applications. Some VB script files are included to demonstrate how to use the XYSoapClient.dll to access the SessionService.

The SessionBrowser program can be used to view data stored in the SessionService, it has to be able to access the session data file used by the SessionService. The configuration file of the SessionBrowser specifies the url of the SessionService and the full path of the session data file.

The Load Sessions button causes all session data from the SessionService to be loaded into the memory of the SessionBrowser. So the SessionBrowser will hold a snapshot of all session data after you click this button. You can refresh the data by clicking this button at any time.

You can selete the session you want to view by using the Session ID List drop down. Then you can use the Item Key List drop down to select a session object belonging to the selected session.

It is also possible to change session data from the SessionBrowser. If you modify the Session Data edit box and click the Set Data button, the modified data will be saved into the SessionService! Needless to say, you need to be very careful when using SessionBrowser to access production data.

To create a new session object in the SessionService, just enter a new session id string, a key string, and the data string, then click the Set Data button.

The Search Text button is used to find session data that contains certain strings. If you enter a string into the edit box to the left of the button and click the button, the session object that contains that string will be displayed in the Session Data edit box. If you click this button again, it will display the next session object that satisfies the search condition, etc.

There are other things you can do with the SessionBrowser. For example, you can extend or shorten an existing session's expiration time.

Typically, the SessionBrowser is used on the same server as the SessionService. It can be run on a different machine if the folder containing the session data file is shared. You need to change the SessionBrowser.config file in order to access a session service running on a different machine.

Finally, I need to emphasize here that the session web service described in this article should not be made available to untrusted applications. Thank you for reading my article. Please visit my home page for my other tools and articles.

Recent Updates

Mask Your Web Server for Enhanced Security

2008-01-08T01:19:00.001-08:00

Introduction
Masking or anonymizing a Web server involves removing identifying details that intruders could use to detect your OS and Web server vendor and version. This information, while providing little or no utility to legitimate users, is often the starting place for crackers, blackhat hackers and "script kiddies". This article explores some ways you can minimize the risk of such detection. Most of the following examples focus on Microsoft’s Internet Information Server (IIS), since it has been most widely lambasted for its vulnerabilities, but some Apache detection countermeasures are also covered. While IIS users probably have the most vested interest here, server anonymization is relevant to anyone responsible for administering a Web server.

Crackers Start Here. Shouldn’t You?
Let’s look at it from the attacker's point of view: Security vulnerabilities tend to be dependent on software vendor and version. Blind probing might lead to further requests being denied or a system temporarily taken off line. Knowing Web server details greatly increases the efficiency of any attack. If an attacker can target exploits, the chances of successful cracking prior to detection increase significantly. Script kiddies can leverage canned, newly-discovered exploits to do more damage faster by targeting hosts with recognizable signatures. A self-identifying system invites trouble.

Port80 Software has developed an IIS server module called ServerMask to combat the majority of issues explored here for the Windows Web Server.

The Server Header Tells All
Most Web servers politely identify themselves and the OS to anyone who asks. Using a network query tool like Sam Spade or this Header Check, you can discern the HTTP Server header. Just request a Web site's home page and examine the resulting HTTP headers or "banners" sent back by the server. Among them, you will likely find something like this:

Server: Microsoft-IIS/5.0
There is not much mystery here. Apache's default settings make it no less identifiable: Server: Apache/2.0.41-dev (UNIX)
You can remove or obscure this HTTP Server header in a variety of ways, depending on your platform. Apache users can use the module mod_headers. IIS users can install IISLockDown and use the configuration option in URLScan's INI file for removing or replacing the header. Be careful with URLScan if you are using Cold Fusion application server- the way the current version replaces the Server header wreaks havoc with CFM pages. In fact, removing the header is the way to go when using URLScan, since if you try replacing the header it moves to the bottom of the header order- which pretty much gives away that you are running URLScan on IIS.
Unsightly File Extensions
Displaying file extensions like .asp or .aspx in a site is a clear indication that you are running a Microsoft server and, in general, hiding file extensions is a good practice to mask the technology generating dynamic pages. You can change your application mappings (.asp becomes .htm or .foo, etc.), but such one-to-one mapping can make mixing server-side technologies painful and does nothing to alleviate headaches during site migrations. Doing without file extensions altogether is an even better idea, not only for security but also for ease-of-migration and content negotiation. Apache people will want to take a look at mod_negotiation. Watch out, though, for the Content-Location header in the server's response, which can give away the file extension that is not shown in the URL. You might have to suppress this header separately using mod_headers. In a similar vein, Port80 is working on a solution to allow file extension hiding in IIS.

Half-Baked Cookies
The ASP session ID cookie, used by the Session object to maintain client state, is another dead giveaway:

Set-Cookie: ASPSESSIONIDQGQGGWFC=MGMLNKMDENPEOPIJHPOPEPPB;
You can disable ASP Session State so that this cookie is not placed, but you lose the convenience of using the Session object to maintain client state. You could also create an ISAPI filter to change the names of any session ID cookie. On the other hand, ASP sessions are resource intensive, and turning them off improves the performance and scalability of your ASP application, while also helping to anonymize your server.
Send These to the Recycle Bin
WebDAV: Another way of identifying Microsoft servers is their implementation (from Windows 2000 and IIS 5.0 on) of WebDAV - the HTTP Extensions for Distributed Authoring and Versioning. WebDAV itself is not unique to Microsoft or IIS; it is a proposed standard (RFC 2518) with an IETF Working Group. Microsoft's WebDAV support, however, adds a lot of information to the headers sent back by the server, especially when an HTTP OPTIONS request is made. If you are not using WebDAV (to support Outlook Web Access or Web Folders, etc.), you can disable it entirely by editing the registry or by using IISLockDown and URLScan.

Public Header: Certain Web servers betray their identity by displaying the Public header in HTTP responses. Few popular Web Servers send this header in response to OPTIONS requests (while almost all respond with the similar Allow header). The presence of Public is a good indication you are connected to either an IIS box or Netscape Enterprise 3.6. The Public header can be removed with a custom ISAPI filter (IIS) or NSAPI plug-in (Netscape).

Integrated Windows Authentication: IIS users should not rely on "Integrated Windows Authentication"- especially not as a way of hiding anything on the server. This method betrays the very secret it would keep, since a script or visual hacker can identify the Windows box by means of the WWW-Authenticate headers sent by the server. When a file or directory is protected by NT Challenge-Response authentication, one of the authentication headers contains the string "NTLM" (NT LAN Manager)- a Microsoft-specific form of HTTP authentication.

Get Your Headers Straight
The number and sequence of your HTTP headers and the presence or absence of certain platform-specific headers provide handy ways for more sophisticated hackers to fingerprint your Web server. A relatively unexplored area of server profiling, this will become a more common exploit as administrators start to implement countermeasures against obvious HTTP vulnerabilities like the Server header. For IIS users, a custom ISAPI filter can alter the Microsoft-specific header order or sequence to emulate, say, a default Apache installation. Apache users can accomplish any header order emulation they wish by experimenting with the location and order of Header directives in mod_headers.

Whose Default is That?
Default messages, pages and scripts of all kinds often contain clues to server identity, and these should be removed or modified accordingly. Software behind the Web server often bubbles error messages back through the HTTP request/response cycle, and customized HTTP errors can mask application server, database server, Web server and OS identity. For IIS, CustomError makes it easy for developers to deploy custom 404 and other HTTP error pages. This article shows how to implement custom HTTP errors in Apache. Avoid this on a development server, since, when done properly, it prevents database and server-side scripting errors from being seen- making it tough for developers to debug their applications! Remove or hide any Web or application server administration pages, scripts or documentation installed under your server's Web root, and make sure to replace those default home pages.

We Don’t Need No Extra Services
Beyond the HTTP service itself, many computers used as Web servers host a number of other network services. Perhaps the most common are FTP and SMTP. As a general security rule, try to avoid running these services on your Web server. In particular, avoid the default FTP and SMTP services in Microsoft IIS. Despite the convenience of integrated services, there is no reason to have Web, FTP and SMTP services interlinked. This is not an issue for Apache, since the Web server is not associated with FTP and SMTP services through a common administrative service. If you do use these services, be aware that they will advertise your IIS server's identity.

When a connection is established with an SMTP service, the recipient server sends a human-readable greeting to the client, the "SMTP banner". What the SMTP banner displays has no effect on e-mail service, but, like the HTTP Server header, it divulges details about the software running on the box. The default Windows SMTP service exposes such information. To find out how to change the SMTP banner, check here.

The default Microsoft IIS FTP server also presents a known banner. Since modifying the FTP banner is a more involved process than modifying the SMTP banner (plan on hacking several system DLLs), your best bet is an alternate FTP server like RhinoSoft's Serv-U FTP server that can display any text message in the FTP banner. As an added bonus, third-party FTP servers like Serv-U are more configurable than the IIS FTP service when it comes to security measures like assigning users their own login directories.

Unsanitary Inputs
Many platform-specific exploits use complex URL strings to gain access to a shell or CGI program, from which a hacker can easily get a directory listing revealing the OS’ default file structure. Once a shell or CGI program is hijacked and the file system revealed, the door is wide open. The best defense against this trial-and-error exploit is a user-input filter or "sanitizer" that removes unacceptable characters (such as meta-characters and their various possible encodings) from user-supplied data. For IIS, the current standard is IISLockDown/URLScan. A new generation of application firewalls extend this protection to the application layer behind the Web server. In the Apache world, user input sanitizing is traditionally the responsibility of CGI authors. Here is the classic CERT article on the topic, with examples in Perl and C. If you are setting up a new box, consider changing the default file structure as well. Input sanitizing and rearranged file structures do double duty- helping to disguise the box and neutralize common exploits simultaneously.

Combing Through the Stacks
Even when all telltale signs are removed from your Web server's application layer, there remain detection weaknesses at lower network layers. Any server with a network connection has a network protocol stack subject to being scanned and identified. The best stack scanners (like NMAP from insecure.org) can ID most operating systems by using a variety of techniques to fingerprint the system's TCP stack. OS-specific IP stacks are also vulnerable to detection via the Internet Control Message Protocol (ICMP), used by the popular Ping utility. Good resources on ICMP vulnerabilities can be found here. The first line of defense against these kinds of network scanning vulnerabilities is a good, well-administered firewall. However, careful network analysis can still identify a box by examining the packets a firewall must permit a Web server to pass through in response to HTTP requests.

Netcraft is Watching
Take a look at the "What's that site running?" tool on Netcraft. If you point the site profiling tool at your own Web site, it will probably correctly report both your Web server and OS. Changing your HTTP Server header will cause Netcraft to report a false value for your Web server- or just "unknown" if the header is completely removed (the change is not immediate, as Netcraft caches results for a time).

Still, your OS will probably be correctly identified- even behind a good firewall. To get Netcraft to report your OS as "unknown", you will have to tinker with some of your default TCP/IP settings, such as the receive window size (RWIN), the maximum transmission units (MTU), the maximum segment size (MSS), and/or the IP header time-to-live (TTL). Altering these settings will affect your server's performance in diverse ways, depending on network conditions, so considerable care should be taken when changing these defaults. In the hands of a skilled network administrator, however, this strategy can be an effective countermeasure to information leakage through stack scanning.

Let's Be Careful Out There
No combination of detection avoidance succeeds in completely anonymizing your Web server- just as no combination of firewalls, IDS, and other security countermeasures can completely defeat a skilled and determined cracker. Like server hardening, server anonymization can help defeat the majority of would-be attackers. And like all aspects of network security, it’s a never-ending battle to stay ahead of the bad guys.

License

Technical Solutions

More questions

Applications of AI

Branches of AI

Basic Questions about Artificial Intelligence

eDirectory Authentication using LdapConnection and custom certificate validation

Source Code Line Counter

Docking Toolbars in Plain C

CWaitingTreeCtrl

Quick Print

Cell blink for DataGridView

A Class to Save and Load Listbox Data

Using virtual lists

Clipboard backup (Visual C++)

Copying a Bitmap of the Client Area

Data Drag is coming to town... the drop target

Time Management for System Administrators

The Test/Code Cycle in XP: Part 1, Model

Revving .NET Remoting

ASP.NET Auto Event Wiring

Concatenating Strings Efficiently

C# and beforefieldinit

New features in C# 2.0

Debugging Unicode Problems

Plug-ins and cast exceptions

Binary floating point and .NET

Decimal floating point in .NET

Reading binary data in C#

Multi-threading in .NET: Introduction and suggestions

Strings in .NET and C#

Constructors in C#

Delegates and events

Parameter passing in C#

ASP.NET Validators Unclouded

Safely Storing Business Objects in Session State

A Session Data Management Tool

Mask Your Web Server for Enhanced Security