New Database Support

We are investigating supporting MySQL and Oracle as managed databases for the nHydrate Entity Framework platform. Please take the following survey to let us know how you feel.

Take the Survey

Database Generator

There is a new article (http://bit.ly/h0Pq6z) on how to use the database generator. This describes the issues with the current state of changing production databases and how we try to address these issues with database tracking. Learn more on how not to use your database as your model and find out the benefits of model driven development.

Deep Dive: ADO.NET Generator

We have released a deep dive article into the ADO.NET generator. This is the most mature of the generators, used for 6+ years in many projects. The article is a little long but still not all-encompassing. It merely touches on many features of the generated framework. There is a lot of functionality there. http://bit.ly/g9UhLL

Generator Library

The new install has no generators in it. We now have a Generator Library. This allows you to download just the ones you want. Now that there are 14 generators and more coming, it has become quite confusing as to what you should use. Now you can download only what you need. Coming soon will be wizards and templates as to how you should generate and dependency information. When you install 4.0.0.186, the first time you open a model file the Generator Library will pop-up. You can also display this screen anytime you wish from the settings dialog on the VS.NET menu.

4.0 Version

The new version is posted. The 4.0 version has a lot of fixes and new features. A lot of base functionality has been added to the Core assemblies. The generated code has been refactored a bit. Also many user requests have been incorporated into the new framework.

Select Commands

There have been some questions lately about select commands. Here is some additional information on how to use them and create custom select commands. If you have custom ways of selecting data that you just cannot fit into the nHydrate framework, read this. http://bit.ly/e0ZpnN

Table Auditing

There are times when you really need to keep track of your data over time. Table auditing allows you to track all adds, updates, and deletes to your data and provides an API for getting change sets. http://bit.ly/h7ZgGQ

Concurrency

If you have ever wondered how concurrency works with nHydrate read here http://bit.ly/gAguE2

Does NHibernate save time?

NHibernate does provide code generation and does a good job for what it is. However nHydrate allows you to build mocks and APIs faster with less hand-written code. See our comparison by following the link below. We are striving to minimize the code you must write and test to get an application out the door. There is a new Entity Framework mocks layer and EFDAL Interfaces layer that allow you to hand-write or use pre-generated mocks to write unit tests. This is very nice as EF does not provide these services out of the box. You get a strongly-typed interfaces layer from your model and it always stays in sync with your real DAL and your database via the installer project.

http://bit.ly/baO1Td

Features of the nHydrate DAL and Entity Framework generators: Part 1 Auditing

Source for this Article

The core of this article will be to explain the auditing features available in nHydrate. Providing the ability to audit database changes is a useful feature for many systems. As such, nHydrate has provided a framework that makes implementing an auditing solution for your project pain free. The examples provided in the article will work on both the traditional nHydrate Data Access Layer (NHDAL) as well as the nHydrate Entity Framework Data Access Layer (EFDAL).

As a precursor to this document you may want to check out the following articles
NHydrate Step-by-Step
Entity Framework with nHydrate
Why implement an Entity Framework based DAL in nHydrate?

What’s in the Model?
Let us start by taking a look at the properties in the model that deal with auditing.

Audit Field Names

The first is on the database node of the nHydrate model. Here you can identify what names you want to apply to your audit fields. These fields will be added to entities that specify they wish to be audited. The following properties are provided: CreatedByColumnName, CreatedDateColumnName, ModifiedByColumnName, ModifiedDateColumnName.

Table Level Audit Settings

For each table in the model, nHydrate provides the ability to turn on or off table level auditing. There are three settings to consider

• AllowCreateAudit - This is a true false setting. When the value is true the table will have two columns placed on it: created_by and created_date. These fields are set the first time a user inserts the record.
• AllowModifyAudit - This is a true false setting. When the value is true the table will have two columns placed on it: modified_by and modified_date. These fields are reset each time a user updates the record
• AllowAuditTracking - This is a true false setting. When true it identifies that a new database table will be created to hold historical audit information.

What’s in the API?

Although the generated frameworks for nHydrate DAL and Entity Framework are slightly different. They both provide means for implementing the auditing features.

In the below examples pay particular attention to the following:

• First we will identify how to set the identity of the user performing the changes. This will allow the framework to setup the modifiedby and createdby columns without requiring the developer to set it on every object.
• There is also a convenience method added to every object that allows you to pull back a history of the modifications. In this way it is very easy to manage rollbacks or present object histories from the API. The convenience method used in this example brings back all audit records. (NOT SHOWN) This method has been overloaded to deal with large audit sets.
• instance.GetAuditRecords() - All audit records for the instance
• instance.GetAuditRecords(int pageOffset, int recordsPerPage) - Paginated records for the instance
• instance.GetAuditRecords(int pageOffset, int recordsPerPage, DateTime? startDate, DateTime? endDate) - Paginated records for the instance between dates. recordsPerPage=0, pageOffset=0 : returns all records between the dates.

nHydrateDAL - Example
// Add a couple of customer objects. You will notice that during the creation of the customer collection
// we pass the modifier. We are also not setting modified_by, modified_on, created_by or created_on
// fields. These are implemented by the framework.
CustomerCollection customerCollection = new CustomerCollection("User14");

//Create a simple customer
//When persisted create record will be added to the audit table
Customer simpleCustomer = customerCollection.NewItem();
simpleCustomer.Name = "Simple Customer";
customerCollection.AddItem(simpleCustomer);

//Create another customer
//When persisted Create record will be added to the audit table
Customer customer = customerCollection.NewItem();
customer.Name = "Test Name 1";
customerCollection.AddItem(customer);

//Persist both customers they will both have the modifier or User14
customerCollection.Persist();

//Update the name. Updated record will be added to audit table
customer.Name = "Test Name 2";
customerCollection.Persist();

//Lets look at what the create a modify produced
//Retrieve customer from database that we just saved.
Customer auditedCustomer = Customer.SelectUsingPK(customer.CustomerId, "User15");

//Write Audit Records. There will be two records.
//The first record will represent the creation.
//The second record will represent the modification.
foreach (CustomerAudit customerAudit in auditedCustomer.GetAuditRecords())
{
Console.WriteLine("AuditDate: " + customerAudit.AuditDate.ToString());
Console.WriteLine("AuditType: " + customerAudit.AuditType.ToString());
Console.WriteLine("CustomerId: " + customerAudit.CustomerId.ToString());
Console.WriteLine("Name: " + customerAudit.Name);
Console.WriteLine("ModifiedBy: " + customerAudit.ModifiedBy);
}


EFDAL - Example
Guid createdCustomerID = Guid.Empty;

// Add a couple of customer objects. You will notice that during the creation of the ObjectContext
// (AuditExampleEntities). We provide a context startup object that specifies the modifying user
// We are also not setting modified_by, modified_on, created_by or created_on fields. These are
// implemented by the framework.
ContextStartup user14Startup = new ContextStartup("User14");
using (AuditExampleEntities context = new AuditExampleEntities(user14Startup))
{
//Create a simple customer
//When persisted create record will be added to the audit table
Customer simpleCustomer = new Customer();
simpleCustomer.Name = "Simple Customer";
context.AddItem(simpleCustomer);

//Create another customer
//When persisted Create record will be added to the audit table
Customer customer = new Customer();
customer.Name = "Test Name 1";
context.AddItem(customer);

//Persist both customers they will both have the modifier or User14
context.SaveChanges();

//Update the name. Updated record will be added to audit table
customer.Name = "Test Name 2";
context.SaveChanges();

createdCustomerID = customer.CustomerId;
}

//Lets look at what the create a modify produced
using (AuditExampleEntities context = new AuditExampleEntities(user14Startup))
{
//Retrieve customer from database that we just saved.
Customer customer = context.Customer.
Single(cust => cust.CustomerId == createdCustomerID);

//Write Audit Records. There will be two records.
//The first record will represent the creation.
//The second record will represent the modification.
foreach (CustomerAudit customerAudit in customer.GetAuditRecords())
{
Console.WriteLine("AuditDate: " + customerAudit.AuditDate.ToString());
Console.WriteLine("AuditType: " + customerAudit.AuditType.ToString());
Console.WriteLine("CustomerId: " + customerAudit.CustomerId.ToString());
Console.WriteLine("Name: " + customerAudit.Name);
Console.WriteLine("ModifiedBy: " + customerAudit.ModifiedBy);
}
}

What’s in the database?
Within the database, additional columns are added to the tables when AllowCreateAudit or AllowModifyAudit are set to true. Taking the customer table as an example, we see the existence of CreatedBy, CreatedOn, ModifiedBy and ModifiedOn.

The next thing that you will notice is a new table has been created in the database schema. This table is where the audit records are kept. This is a result of specifying AllowAuditTracking to true on the customer table settings

Database Diagram for Audit Model
Now we can look at the results from running our code. Within the database we will notice that auditing fields have data for both of the customers we added. This occurred without the overhead of a developer expressly setting them on every object that is stored. We will also see that the audit records have been established in the __AUDIT__Customer database table.

Customer and __Audit__Customer Results.


Miscellaneous

• The framework can set ModifiedDate, CreatedDate and __insertdate as UTC or Local values. Depending on the UseUTCTime setting, all of these times are established on the database server.
• Manually setting the ModifiedDate or CreatedDate will override the values set by the framework.
• The database stores an integer to identify the audit actions. 1 = Create, 2 = Update, 3 = Delete
• Fields that are of type text, ntext, or image are not available in the audit tables.

Why implement an Entity Framework based DAL in nHydrate?

Although nHydrate is only a year old from an open source perspective, it was originally created over five years ago on the .Net 1.1 framework. During this time, nHydrate sets out to enhance the data access experience using a Model Driven Architecture (MDA) approach. Using this approach required us to provide a model at a higher level of abstraction than a typical data access layer. Our core focus was to ease the pains of evolutionary database design while maintaining a robust data access layer (DAL).

The nHydrate team has always been dedicated to staying up with the latest Microsoft technologies. As a team, we do not wish to be stuck on an out of date technology. Over the past couple of years, we have watched carefully as Entity Framework (EF) has evolved. It was not until the release of 4.0 that we felt it would be viable for us to provide an EF solution. The advent of a viable Entity Framework architecture, also allowed us to demonstrate the flexibility of a MDA based application. Using the same model, developers can generate very different DAL solutions.

At this point I feel it is important to identify our continued dedication to the current Data Access Layer (DAL). There are many applications in production today that depend on this architecture. Entity Framework 4.0 is not a good solution for many of them. Specifically, applications that wish to provide a DAL that consists of hundreds of tables. For these reasons the current solution is core to what we do and must continue to be a focus.

Why choose to use nHydrate with Entity Framework?

To get a good answer to this question you must look at our past. The majority of our current architecture is built on Dataset technology. When we originally investigated this technology two things became evident. First, the models that drove the dataset were created at the project level. We felt that an MDA based application should have a model at the solution level. Second, there were several improvements that could be built on top of this architecture. Anyone who looks at the nHydrate architecture today would verify that it has many advantages over strait datasets. Entity Framework is not an exception to either of these realizations.

Taking a more concrete look at the current EFDAL release the following items are addressed:

• Evolutionary database management – The current database product has been adjusted to capture the needs of the Entity Framework model. The nHydrate team is bringing evolutionary database concepts to an Entity Framework solution. For those using nHydrate today, another important realization should be made. The EFDAL and the nHydrate DAL can on the same database. Making a mixed implementation possible today and a migration path available in the future.


• Type Tables – Managing enumerations in your code and database can be a cumbersome task. nHydrate solves this by allowing users to define type tables that present themselves as enumerations in the DAL. Example: In the Order table we need to persist an Order Status column that represents one of the following values Created, Packaged, Shipped, or Received. The management of this enumeration in the code can be handled by adding a type table to the nHydrate model.


• Stored Procedure Generation – All insert, update and delete functions are automatically mapped to stored procedures.


• Property Level Eventing – The entity objects are provided with an event model that allows users to hook into specific property changes.


• Explicit partial class generation – If you have used our generator in the past you would notice that we like to generate an editable partial class file as a parent to the code generation file. In the below example the Customer.cs file contains a partial class that is used to extend the generated Customer object in Customer.Generated.cs file.
  
  
  
  


• Convenience property for derived table – An example of querying the derived shape circle is below.
  
  


• Implicit Load - Automated calls to load method the first time a property is walked.

To learn more please download the sample project at http://bit.ly/c9PmCz. Also, follow us on twitter or join our linked in group. More documentation and examples will be provided through our blog, the code project, CodePlex and You Tube. The best ways to monitor these events is to follow our twitter feed or subscribe to the LinkedIn group. The side of this blog provides a link to these social networking sites.

Where are we going from here?

The majority of this question will be answered by the users that provide us with feedback. However, there are a few things that we must support. First, we must provide a solution that provides a transition path between the competing data access layers. Second, we have a short list of features that we wish to support on Entity Framework over the next few weeks:

• Auditing – Modifier established at context level, Creation of audit tables automatically filled through create, update and delete operations.


• Bulk operations - update, insert and delete


• Generation of a RIA services project.

Performance Tests

There have been some questions on performance of the DAL and we had no stats to give. So we have out together a little test to at least get some gauge.

The test was performed on a AMD 3GZ Quad core machine with 8GB of RAM. The database had three tables: USER, USER_TYPE, and JOB. There were 48,652 users and 38,949 jobs. Each job has a parent user. The test was broken into 5 parts that were each run on a thread. There were 22,906 unique last names. There were no defined indexes except for primary keys and the user Id foreign key on job for its parent user. The tests performed were as follows.

• Aggregate count of users with a last name
• Select a list of users by last name
• Select each user by primary key
• Walk to related jobs for each user
• Walk to a parent user for each job

The results were as follows. The test was run in two minute increments and used 60-70% of the processor. About 700 records/second were pulled. This was from all operations, which averaged about 630 operations/second. An operation is defined as a database hit. The number of records pulled per transaction was small, so this reduced the overall selection average. 77% of users had less than 10 jobs, so the result set pulled was very small per operation. The time to pull 1,000 records is virtually the same as pulling 100 records. In other tests that were pulling tens of thousands of records at one time the results were retrieved in the 1-2 second range. Indexing of course improves results even more. The tests above were performed querying against last name which was not indexed.

These tests show that the DAL (data access layer) can handle quite a bit of database operations. If you assume that a user will hit the database 5 times / second (an overestimate) then this scenario still admits to handling 126 concurrent users (630/5). More realistically a user hits the database in batches, like loading some on-screen list boxes, client information, etc only every few seconds. So even assuming database access once/second/user still yields several hundreds of concurrent users.

Keep in mind that these tests were done on a sub-$1,000 machine, a desktop with application and database on the same box. In a production environment, you would most likely have a stand-alone database server with proper indexing. This would most likely include a RAID system, which helps tremendously. For further speed enhancement, you might also implement some sort of database load balancing if you are running a large-scale, enterprise application.

To conclude, these results were thrown together with no optimization whatsoever and they still show how performant is the system. In the real-world, you would be using a better machine with some simple and more complex optimizations. For these reasons we feel that the data access mechanism of nHydrate performs very well indeed, for the domain in which it was designed and runs.

VS.NET 2010

We have finally finished a first version of a 2010 plug-in. So if you work in a cutting edge kind of shop, you can get started generating code now. Please keep up with the project as we are making enhancements. The next version will most likely handle multiple database schema. This is a feature some large projects need.

Demo Complete

W e went to Orlando and gave a demonstration on nHydrate. There were some very interested people there. Most had not heard of the product, but came out of curiosity. We were pleasantly surprised at the discussion in the room. Every one of the participants actually asked questions, gave feedback, and took notes. There was a direction pushed we had not thought of before. A Clarion developer said that his community really wanted to move to NET but they did not like the generation tools. Apparently Clarion has a lot of code generation and they do not want to give it up. So maybe this can be a way for Clarion developers to transition to .NET.

In the demo we went over ORM and MDA. We created a simple object add and dependency walking. We also showed the aggregates and bulk data operations. The Lambda syntax was quite impressive as was the inheritance table splitting. At the end, we touched on the IoC framework with POCO objects and such.

Many people were interested in the jQuery scaffolding with grids and data interaction. We discussed generating some of the UI in the future with customizable grid and other features. We also had a discussion about where future development should focus. Currently we are focused on jQuery however we may need to focus on Silverlight. This really requires more discussing from users about the best direction to go.

The model will be changing in the near future to add some new functionality like grids, enhanced UI generation and scaffolding and perhaps some other new stuff as well. Feel free to comment on this blog. We are also going to start sending out surveys to get feedback of desired technologies and functionality soon.

All in all we were very happy with the response and look forward to doing it again.

Orlando Code Camp

We will be at Orlando Code Camp in Orlando Florida on March 27. We will be hosting a session on Model Driven Architecture (MDA) using nHydrate. We welcome suggestions and feedback. Michael Knight will be the speaker and we hope to get more people interested in rapid application development using MDA. If you are in the area please come by as this is a free event. Here is a list of sessions.

http://www.orlandocodecamp.com/Agenda.aspx/Sessions

Site Outage

We had a site outage for www.nhydrate.org from Tuesday night until Friday night. The site was down and no one could register because the registration service was down too. This is all fixed now and we have moved everything to a new location with better RAID, backup, uptime, etc. Sorry for anyone trying to access the site or register but we are back up and running now.

Calculated Fields

The newest posted version has support for calculated fields. This allows you to define (or import) fields that are derived from a formula. This was a customer requested feature and it finally made it in there. Also the generated code has been optimized to create smaller binaries. Also the Northwind sample has been regenerated, compiled, and  uploaded to keep in sync with the latest framework.

Shrinking the code

We are currently working on optimizations for the generated DAL. Experiments so far has yielded good results between 6-12% of the assembly has been shaved off. The good news is that the bigger your model (more entities), the better the results. Bigger models get a much better percent shrinkage. One of my tests is a 276 table database and compiled assembly is 12% smaller. These enhancements will be live in the next version.

More Aggregate Functionality

There is now very advanced aggregate functionality. Implicitly walking relationships to pull aggregate data is simple and compile-time checked. For example, say you have a list of jobs and there is a related list of applications for those jobs. Now how do you pull a users that posted jobs that happened to get applications yesterday? The code below shows how to do it. The JobApplicationCollection class has a static method named "GetDistinct" (and also min, max, sum, and count aggregates) that takes a LINQ expression. The statement below reads: pull back a list of job owner user ID integers that have jobs posted and have at least one application in the related job application table where the application was created yesterday.

IEnumerable idList =
JobApplicationCollection.GetDistinct(x => x.JobEntity.OwnerUserId,
x => x.CreatedDate < DateTime.Now && x.CreatedDate > DateTime.AddDays(-1).Now);

This is very powerful syntax and can be used to pull back data from any table (like Job) that is related to the calling collection's base table (like JobApplication). All relations are defined in the model, so there is no joining in code. Simply define any arbitrarily complex LINQ syntax and pull back data. Remember this is all compile-time checked so there will be no surprises (errors) at runtime.

DUCT TAPE DAY IS COMING FOR US ALL

Have you ever heard of the saying "Jump the Shark"? John Hein defined it as: "A moment. A defining moment when you know that your favorite television program has reached its peak. That instant that you know from now on...it's all downhill.” There is a website dedicated to this phenomenon where people can vote on what episode the television program actually “Jumped the Shark”. In the software world I refer to that point as “Duct Tape Day”. As with “Jump the Shark”, development teams can sit around vote on the day it actually happened. But most everyone will agree when it has passed. So what is duct tape day? In a nutshell, Duct Tape Day is a defining moment when you realize that all your hard work, all the hours spent keeping the architecture pure and clean, has been a waste because the pressure to complete the product is forcing you to start duct taping it together. If you have never been on a product that has faced this day, then I commend you. You need to read no further. I feel you are an aberration to be admired by all. But save this page as a bookmark, because I suspect you will come back to it one day.

At this point, you may be expecting me to point out ways to avoid duct tape day. How can we as software engineers ensure the day never comes? If you are looking for that from me then you have come to the wrong place. For instead of trying to avoid duct tape day, I embrace it. I expect it to happen, plan for it, and learn from it. If I am lucky enough to have a project that never sees this day, then my plans are in vain and I have nothing further to learn. However, I have yet to find a consistent way to keep the day from happening.

If you look across the landscape of television programs, you will notice that most of them have jumped the shark at some point. Yet they remain profitable. They still have a manner of success. This is because of the solid base the television program stands on. This base cannot be taken away by tacky new characters or crazy plot lines. The fans of the show have memories and love for the characters that will see the show till the end. In a similar manner most software projects have passed duct tape day. Tacky characters and plot lines can be analogous to unplanned requirements, quick fixes and hacks. Developers however will continue to be loyal to the parts of the architecture they grew to love.

In software development, building up a base architecture is very challenging. It is a savvy architect who can build a base that other developers believe in and want to continue working with even after duct tape day has past. To meet this challenge you must make quality decision at the start of the project. Professional software architects love to learn new things. They love to read technical papers and keep up to date on the latest trends in software. And worst of all they love to push the latest technology and practices into the products they build. If an architects starts a new project with the attitude:

This project is going to be different. We are going to follow best practices that we have not been able to follow in the past. We are going to use the latest technology. We are going to fix requirements that have been missed in the past.

Be wary, the project is in danger of not being ready when the pressures of business comes knocking. Getting the architecture completed for a project that starts out with this approach is difficult enough. However, this is not even the greatest concern. Getting new ideas accepted and ingrained into the developers daily life is the hidden challenge. When business pressures come, developers will work with what they know and understand. Things that are not ingrained will quickly fall away.

To get my point across we can talk about unit testing. Unit Testing is a great idea and one that developers should strive to have within their application. Most will agree that unit testing should be considered when starting a new project. What the architect must consider is the effort to implement unit tests. Although people say that unit testing will reduce the cost over the lifetime of the project. Nothing comes for free. Unit testing requires an effort up front to get started. It also affects the overall architecture of the code base.

1. What unit test framework should be used?
2. What difficulties does mocking add to the overall framework?
   
   
   1. Can the team use a mocking framework?
   2. Do we need to bring in an IOC framework to implement mocking?
   
   
3. How will we police unit tests?
4. Do we separate developers writing code from the ones who are required to write unit tests?
5. How do we force unit test to run as part of our build process?
6. Will unit testing at the user interface level require us to use an MVC, MVP or other similar pattern?

This is a simple list. But duct tape day is not about getting this list done. Duct tape day is about something architects all too often forget. Duct tape day is about developer acceptance. It is on that day that an architect can determine what he did that made a difference in the developer’s daily life for good and bad. When the pressure comes, developers care about getting the product out. Their loyalty to the architecture is gone. A mentality shift occurs. Anything that helps them in their daily life they will continue to do. Anything that does not will be thrown away, patched together or worked around. That is why I call it Duct Tape Day. Developers take what they know will work and patch a solution together. Architectural decisions that did not help the developer are turned into a massive ball of duct tape. The structure is lost and for the most part the purpose of that architecture is destroyed.

Considering unit testing, I would challenge a development team to ask themselves, are we writing unit tests or practicing test driven development? The key thing to realize about this statement is developers need to get something tangible and immediate from the work they are doing. They need a carrot that is so beautiful it cannot be taken away. They need to feel like they cannot do their job unless unit tests are implemented. A software architect cannot expect developers to write good unit tests just because they tell them to do so. The architect must make sure that unit testing is easier and more accepted than testing the program by hand. If not unit tests are doomed for failure. As pressure mounts they will be ignored. Failed unit tests will not be fixed. The process of writing them will cease. Good idea but too much work will be the statement made by all. It is at this point you can truly realize the extent of the failure.

Other architectural aspects of the application follow the same paradigm. If the architecture does not make the developers day easier, then it is at risk of being destroyed. Planning for “Duct Tape Day” means spending time up front to consider the developer. Think about how much effort will be needed to gain developer acceptance. How will the architecture make the developer’s life easier? This is why I embrace duct tape day. It is the day I realize what I did that developers accepted and ignore.

My Advice: Do not be sad or threatened about the results from duct tape day. Do not love your framework so much that it hurts you to let it go. Use the development team’s willingness to follow as your guide moving forward.

• There is another version.
• There is another product.
• What worked well can be continued?
• What went poorly can be scrapped?

Most of all remember duct tape day is the point when you realize that architecture is not just about satisfying the customer’s needs. Architecture is about satisfying the developer’s needs. If the second is not considered, your product will end up a great big ball of duct tape.