Share This Using Popular Bookmarking Services


Google ads

Recommended

network monitoring software




Programs & websites tailormade for you.

OOP, SOA and thoughts on designing layered architectures 

Thursday, June 14, 2012 8:06:00 PM

This post is my attempt to give words to my thoughts in connection which changing the way I program:

  • I am used to make asp.net websites with lots of data input and output, that more or less just straight passes data into and out of the database, with no more logic or design than using typed DataSets to get the data all the way from the database to the databound controls. And I am used to get products extremely fast, and keep my deadlines because of a very simple design philosophy and efficient programming tools.
  • But now I am starting on a new project, where there can be several front ends – that is: both a website, a rich client and possibly other customer written programs that all need to communicate through a web service to get data or do anything. And the time constraint is still that the first usable version (one platform) should be produced very fast, and then expanded on as the concept is developed (and more paying clients come aboard).

That means, in order to make a system that can be expanded upon, I probably have to design it with all the different layers that I usually did not need, and I am trying to figure out why they are needed! Please bear over with me if I use the concepts/words slightly wrong, and please do comment if you disagree with my observations.

I like object oriented programming, and my first thought was that I should write the business logic layer as part of the domain model, so there for example would be an "Order" object, which was not just a database mapping, but would also contain method like "CompleteOrder()" that could trigger business logic (state changes, updating other records, sending emails etc.). In other words: I just wanted entities/classes describing a domain model to be the only layer that needed to be designed, which could be passed all the way from the database to the UI through an automated web service.

But as soon as you go through a web service, the classes with their methods are lost, and you only have basic data structures with no methods...

I started searching for ways to make it possible, but it seams like the answer is always: Don't do it!

So I now have to look at the layers model again, to see if I can get it to make sense...

What are the potential pitfalls in the different models/designs/layers (or in misusing/misapplying the models/designs/layers) that I have seen or want to avoid?

Wrong goals:
  • Making "dumb interfaces" seams to be the holy grail for architects who want all logic in the business layer – but guess how those interfaces feels to users? Right, they feel truly dumb, by not showing the slightest aid or understanding in the data they present or how they might cross link to other data! Accountants might still kept track of everything using big digits for account numbers, but everyone else expects drop down lists, auto complete/suggestions, validation of input with detailed messages rather than “failed to save”, and in addition to that a beautiful and intuitive user interface design.
    Possible solution: Design your user interfaces for a great “modern” experience, and if a website require client side Ajax functionality then you better design your layers/architecture to deliver the required data to the UI. If you have a framework that can automate the production of intelligent error messages in the UI from other layers then use it, and if not then you have to add it yourself – your user's experience is more important than how easy or difficult it might be to maintain the program, because without users you will not maintain anything!
  • Making a “super flexible systems” where every object can be expanded with random additional attributes. While I might admire such complex systems, they are an UI nightmare to design so they work smoothly (meaning: it takes a long time to code right), and once you have a couple of million records in one table, where 20 lookups are needed to collect all attributes for one object that should be part of a statistic calculation together with 500 other such objects, you might find your UI has become very sluggish... I have added SQL tricks to such a system to get UI response time down from 16 seconds to 1 second, but that means even more complexity to maintain.
    Possible solution: Make a simpler (less flexible) design that fits 90% of the potential customers, and make the system designed so it is easy enough to maintain (by adding new pieces of information to the model as needed by new user/business cases).
Complexity making things harder:
  • Thinking that all these layers of objects will make it "easier to make changes"... If I have to understand and change 3 to 5 layers with each layer using its own DTOs, just to get one more column added to the database and presented in the UI, how can that be easier than passing a row of table data from the database to the UI, where you only need to change the database and the one object? I have never been asked to change the database or infrastructure of a program – the requests I get is that the users need a new kind of information in their UI, such as an extra piece of information on a contract, and this change have to be implemented all the way from the UI to the database, so the more layers the harder it is to add.
    Possible solution: Well, if multiple front ends are in play, then some layering is mandatory, and I am trying to figure out the right amount of structure needed to solve the problem – read the rest to see if I come to a conclusion.
  • Writing data access layer, business layer and presentation layer for something that is trivial (resulting in hundreds of lines of code that does nothing except pass data to the next layer, without even as much as a web service or other boundary or error message handling introduced to justify it, nor any business logic that needs to react to it).
    Possible solution: Even textbooks recommend simpler architecture for simple input/output applications – I guess the trick is knowing when something is no longer simple, such as the involvement of a key object that needs special handling to follow business logic/integrity.
  • An OR-Mapper can generate nice objects, all linked together, stored in many multiple tables, but if you generate a UI list view that needs all those linked objects preloaded and populated, although only select pieces of each linked object is used, it can result in several second waiting time to load everything in. I have seen one programmer's handling was to cache the whole object chain in another memory cache mechanism, but he then forgot to invalidated when a change happened... More and more complexity and workarounds needed, that all follows from the simple wish to treat the database data as nice hierarchy of objects. Oh and the code to generate the list of objects had of course its own specific syntax for that OR-Mapper, many times longer than a simple SQL select-join statement that would produce just the exact needed data for that list view in a fraction of the time.
    Possible solution: For displaying complex lists you need to be able to bypass any OR-Mapper with its complex domain model (with SQL to a plain DataSet, or an OR-Mapper like NHibernate that seams like it can fill custom DTOs from special HQL select statements).
Making objects that are not following object oriented programming principles:
  • An aspx-page is in itself an object, but that does not automatically make anything you write object oriented – one example I saw was to make 3 functions that all used object variables just like old time spaghetti code global variables: one function loaded data, one function initialized the UI drop down lists with possible choices from loaded data and one function bound the loaded data of the record being edited with the UI – but they were called in the wrong sequence (the last two switched around), so no drop down lists showed the correct data from before... Passing local variables between functions would have prevented the switch, or just leaving all 3 pieces in one big function would do just as well – unless a piece of code can be reused another place, why break it up at all? It just makes it harder to track what is happening.
    Possible solution: Keep true to the OO principles – any 'global' variable inside an object must be useable for any function at any time and even internal functions should not have hidden side effects; you can use get{...} to initialize its value on first use, or you can ensure the page life cycle is designed so that all variables are initialized and valid for all stages – and if that is not possible, then make it a local variable that is passed from function to function. Alternative: Use the object's (ListView/Repeater) life cycle events to ensure stuff are done in the right sequence, such as initialization of drop down lists on creation, and binding current selected values on databinding.
  • An OR-Mapper generates objects, but if you don't add business logic to them, then they are no more than fancy (heavy) database records/structures, and you are still thinking with E/R diagrams and tables instead of objects that have responsibilities/functions/purpose – which is visible by the need to make other classes that handle updating/inserting and link integrity.
    Possible solution: Either drop the OR-Mapper or realize that you might be using it for a different purpose than object oriented design (for example you might benefit from a build in validation functionality).
  • If you are writing controllers, handlers and managers to contain your business logic, such as an "OrderHandler" to sum up the total cost of an order minus discount, then those controllers, handlers and managers are the key components you should document in your design – they are your true business layer – while the domain model with its OR-Mapper objects is just a database design/layer.
    Possible solution: The right OOP solution would be to place all this business code as part of the domain model classes (by expanding the code generated by the OR-Mapper's partial classes with your own functions), so that the order class itself would be able to make the calculations from the other objects it is responsible for.
  • Having an “OrderHandler” that does business logic like summing up an order plus correctly calculating discount and tax – and then hide the calculation away from the UI, so the UI have to repeat the same calculation 5 different places in order to present the sub-figures like how much the customer has saved, which makes it extremely complex to change the discount structure.
    Possible solution: If you choose to make a new class with the responsibility to handle other classes/objects (instead of putting the code inside the affected classes themselves), then you also have to use that class everywhere, and you need to give it enough functionality so that it truly can do everything that is needed regarding those objects.
Failing to make a layer when a layer is needed:
  • I am maintaining the website for a system that also has a rich client, which is maintained by another programmer. The two front ends do not have any common layer to go through except the database itself, so business logic like “update that record when this record is updated” can only be done by triggers in the database. It has worked for more than 10 years, and we are actively adding new features, but as we progress towards more complex logic, it becomes clear that it is definitely not the best solution!
    Possible solution: Adding a common service layer, even if only for new features, that could encapsulate the needed logic.
Placing classes in the wrong layer:
  • Example: Defining a business layer and a presentation layer, and then place a business logic thing such as an OrderHandler to calculate the sum of an order in the presentation layer, and then place the sending of a confirmation email in a OrderController in the business layer – this way the two are unable to talk to each other, and resulted in duplication of both the calculation and number formatting.
    Possible solution: Place business logic in the business layer! Then make a separate project for presentation-like utility functions/classes, such as number formatting and sending/formatting e-mails from the backend services, and let both the business layer and the website presentation layer reference this utility project so that the business layer can trigger an email, while both can share formatting code (you probably want the confirmation message on the screen to look like the confirmation email that is being sent, without having to write it twice).
Writing code that is never needed:
  • Writing abstract interfaces for all classes although there is no plan to ever make more than one implementation of each (it just results in two files with duplicate function definitions, making it harder for another programmer to navigate around the function calls to the actual code definition). I have never added a new implementation to an existing interface, but I have many times been adding new data and functions to old classes, which required double work to update their interfaces too. And you will probably not have found the right way to instantiate the correct version of the implementation until you know what circumstance require a different implementation.
    Possible solution: Avoid them unless interfaces are predicted to be of use (or they are required by some other technology such as WCF).
  • If you need several cheap programmers in some far away country to write thousands of line of code just to transfer a drawing of objects into code, and all that work is needed to be done before you can write the first line of business logic that controls how things actually react, then there is something wrong with your selection of tools.
    Possible solution: Use tools that can generate and update simple code such as class definitions, but ensure the generated code can safely be expanded upon with your own functions/logic. Use programmers who put a pride in writing solutions, and who are not paid per number of lines of code.
  • Writing a Data Access Layer that does nothing except calling the same functions on your OR-Mapper/TableAdapters, and thinking that a DAL with several thousands line of code and SQL statements and/or OR-Mapper specific code would be easy to convert to another database, or would make it easy to do database changes without affecting business logic. My assumption: If you have the need to restructure the database, then that need should be based on new business logic requirements – so the goal of being able to change the database structure and keeping the business layer the same appears to me to be to be wishful thinking.
    Possible solution: I suggest that the automated code generation by design tools like typed DataSets and OR-Mappers like NHibernate can constitute 98% of what is required of a DAL, where the tool makes it very easy to makes structure changes, and those tools can even truly make it easier to switch database since the interface remains the same (the last 2% is connection string handling and possible special mapping).
Forgetting to design for security:
  • Access permissions needs to be thought through, as it can impact model/functions – for example you don't want one user to be able to see another user's data just by typing a different database key in a URL/cookie/hidden field/post for a request, so a framework for validating and identifying the user is needed, and the user identification needs to be passed along with the requested key to the backend (web service / business layer / DAL) to ensure that despite requesting an object by a unique key, it is only served if the user actually is allowed to see it. That might result in either putting an owner id into all tables (and selecting objects using both primary key + owner id) or decorating all select statements or business logic with lookups to ensure the owner is correctly linked to the table.
    Possible solution: Design security limitations into your business logic and database from the start, and make it visible in your domain model.

Considering all those pitfalls, what are the arguments to do, to combine or to skip writing each layer?

Can the presentation layer be skipped?
  • It might sound like an odd question, but there is tools that can automate the generation of UIs based on a data model – but my experience is that those break down when you want to add the first new UI feature for a better user experience. Even when it is possible to extend such tools with new features, the work required is often bigger and more complex than writing the whole UI manually.
  • Some might try to just have simple aspx pages, that each are totally independent of each other, but you soon find the need to abstract things like menu structure and access rights out, and you might quickly find the need for lots of extra utility functions, so in the end you will have a presentation framework no matter if you designed it consciously or not.

Conclusion: It is better to decide which framework to use (or design one yourself) and really thinking about this as a presentation layer that needs some design thought, rather than just letting it grow on you like spaghetti. Ensure the infrastructure needed for making a great UI is in place, and writing the actual forms becomes much easier.

Could the business layer be skipped?
  • If it is a simple data input/output form, then the simplicity of going straight from the UI to the database might be rationalized based on time requirements. But before you can do this decision, you need to have identified all those objects that really need some business logic, and you need to be damned sure that you are not violating any potential business logic in the future, otherwise you will end up basing your business logic on triggers in the database...
  • If it is the business layer's responsibility to maintain integrity between linked objects (such as updating a total in an Order object when an OrderLine object is added or removed), then that is a vital ting to design properly – and no matter how basic your solution is, you might as well group the result together and call it a business layer.

Conclusion: You need to design what constitute your business logic before you can safely bypass it, and instead of having *handlers and *controllers spread around your code like cheese on spaghetti, you should make an effort to design this logic so that it has clear responsibility and is always used all those places where it is needed – and no matter how simple or how complex a design you choose, that ends up being a business layer.

Can the business layer and the service layer be considered as the same thing?
  • Not if: You are letting business logic objects with functions through to be used directly by the presentation layer (although it is not recommended to do that).
  • Not if: Your business logic is in functions in a domain model, which forces you to write a service layer to map the plain web service API to the right domain model objects.
  • Possible if: Business logic only consist of simple transaction script pattern functions in the service layer that directly updates the DAL (that is: no business logic based on functions in domain model classes).
  • Maybe not if: You are making a layer client side, that abstracts weather or not calls go through a web service or as direct function calls to another assembly. Note: That would be a client side service layer abstraction, so that in effect you have two service layers – one server side and one client side, and the server side layer could still be considered to be the same as the business layer.
  • Maybe not if: You have a business object that calculates stuff like summing up lines of an order, which you would like to have running client side also (in a WinForms application or maybe even in JavaScript) to avoid round trips to the server for each individual order item added. That should be in a separate project (separate from server-only business logic) that can be reused both in the service layer and in the presentation layer. Note: The service layer could still be the business layer, only the shared part is passed on to the other assembly.

Conclusion: Only possible to combine these layers in a specific scenario, as outlined by the limitations above. But might be worth trying...

Should additional layer(s) be added on the client?
  • You could have a standard library that encapsulate the web service communication of endpoint and security, and possibly configure it to bypass WCF if used on a website (could be an advantage if you suspect that you might move the website to the cloud, separating it from the business layer and database). You could expand such a standard library with infrastructure to handle lost communication handling such as a “reconnect or exit” dialogue for a rich client. But it must be an optional library, meaning that others should be able to write programs to communicate with the web services even without that library.
  • You could also have the whole Domain Model with functions and business logic in a standard library for rich clients to use, but the web service boundary would complicate things enormously, and the web services would need to verify that the clients have not altered the business logic. Such a model could also obscure potential speed problems, as either too many objects needs to be preloaded, or too many small objects would be late loaded. So while I would like to avoid making extra DTO classes, making optimized DTOs suddenly seams like the only safe choice, and actually much easier/faster to program. Note: Those DTOs still need to be properly defined on both end of the communication line, but it is easy to import common class definitions in both layers once they are made with any tool.

Conclusion: The first idea – to make the client unaware of how it communicates with the service layer – could potentially save some work, and provide a framework for increased stability. But since stuff passes through it transparently (it's classes and methods should be a 1 to 1 representation of the web service endpoints and functions), it is in a way more a solution to the presentation layer problem of handling broken connections, and not really a layer at all... (oh, who am I trying to kid – I'll have to code it no matter what I call it).

Can the business layer and the data access layer be combined?
  • If you add your business logic to the classes generated by an OR-mapping tool, so you have implemented a domain model with both data and functions, then yes. I think that could be true OOP programming. Although there would probably still be some things, such as generating connection strings, which should be isolated for itself as a glue between the objects and connecting to the underlying OR-Mapper/database.
  • But you would have to either expand the domain model to include all the DTOs needed for special cases, like solving the heavy list view mentioned earlier – or you would have to make a lazy loading senior Domain Model object with a function to generate such fully populated DTOs for junior objects, bypassing the heavy model while still having glue to find the matching Domain Model objects when later needed.
  • For OOP people like myself, it is very tempting, but it breaks down when passing a web service – everything needs plain DTOs, and all class functions need to be rewritten as web service methods that all need a reference to which object it needs to instantiate before calling the class function. That it not just double work – that is two very different viewpoints on the same data!

Conclusion: For my current need (an application with a service layer), this OOP design seams like it generates extra work without giving any benefits in return. But if I was writing a simple website from scratch, with no other front end ever planned, then I would be very tempted to go this route!

Can the data access layer be skipped?
  • As earlier mentioned, I think 98% of it can be done using tools, so while data access, objects/classes/tables and databases is obviously needed to be designed (it is the core structure of the whole system), coding it should most of the time be as simple as pressing a button after making a design and some select statements.
  • In some circumstances, such as requiring support for multiple databases from day 1, or complex mapping between a domain model object to multiple database records, you really do need to write at least pieces of a DAL yourself.
  • If you model your DTOs with the same tool as the DAL, or otherwise transfer results directly from the DAL to the UI, you might want to split the automatically generated DAL code in two parts, where all layers of your system knows all the structure/models, but only the business layer has access to the part of the DAL that actually perform updates to the database (or only the business layer has access to the glue such as connection strings that would make update commands work).

Conclusion: Big time savings is possible with tools, but only if the business layer is responsible for correctly using a generated DAL (such as maintaining link/relation integrity where needed), or if it is safe for simple data input/output situations to make the UI bind directly with a generated DAL. But you would always have something that you could group under the heading of DAL.

Can a database be transparently served as a web service (moving all layers to the rich client)?
  • Tools like Data Abstract and WCF+Entity Framework both gives more or less automatic ways to expose the database as a web service in such a way that your client would have a DAL that communicates with that web service instead of a database. In that way you can also make a true OOP domain model inside your client applications.
  • But to make multiple clients behave according to a common business layer, that layer has to be in the form of a shared assembly/code that is used by all clients.
  • Alternatively, tools like Data Abstract makes it possible to have scripts of business rules run on the web server. While it is a smart design, in a way that feels similar to implementing business rules with triggers inside the database... I have to investigate it further to see if the result is more complexity or more simplicity.
  • Exposing an interface that allows clients to perform selection and updates based on primary keys, could be a potential security nightmare: How do you prevent one client from accessing data belonging to another client, simply by trying to request other primary keys? Even worse, if you want to expose the web service to clients to make their own add-ons, how do you add code to verify that every single request is valid to perform?

Conclusion: In an environment where every client application is fully controlled and thoroughly validated to be secure and trusted, this is a valid model, that could remove all worries about designing a web service, since the web service simply is your database - and I know one programmer who is using this model successfully. But if the business layer code cannot be shared between all client applications, or if clients potentially need direct access to the web service, then I think the web service would have to be modified so heavily with security checks that the automatic generation tools might start to work against you, meaning you could be better off by designing the web service yourself.

 

A though about function versus structure

From studying another field I have learned the datum that: function is senior to structure.
Applying that datum to modeling/design, I get:

  • Structure (meaning: real world objects from the problem domain) has to be understood and decided upon (and it is often the easiest place to start leaning about a new problem domain), but it is the functionality (such as use cases) that tells if the structure is good, just usable, lacking details or horribly complex.
  • Nobody forgets to create tables in the database, but many (including myself) forgets to design how the structure is supposed to be used/function, resulting in a random patchwork of “*handler” classes. A clear design of how things are supposed to function is an absolute requirement, in order to be able to maintain or expand a system with ease!
  • You have to decide on the proper framework that supports all your needed functionality from the start of a new project. Even in the presentation layer there is lots of decisions regarding how number & date handing in different countries are handled, how security is implemented, how one screen is linked to another screen, or how data updated one place might be reflected another place, how broken connections in rich clients are handled – functionality that need thought, design, implementation (additional structure) and documentation, before you can start to drag and drop UI elements onto forms. If you forget it, you end up with a mix-max of duplicate code, or the next programmer on the project will not notice your added fixes so he'll code his own.

Conclusion & lesson learned:

I wanted to avoid the complexities with many layers and DTOs, but my analysis sort of says that the layers will appear no matter if you design them or not – the result will just be more messy if you don't plan them from the start.

There is possible time saving shortcuts, such as using a tool to generate the DAL, maybe using a modeling tool for easily making DTO classes, and there is frameworks that can ease some of the logistics. All those can help.

Combining the service layer with the business layer as one layer is something I would like to try – that is: designing an architecture that follows the prerequisites that makes such a solution a viable alternative (although it means not making a proper OOP domain model, OOP is still used on many other levels with big benefits).

But the biggest lesson that I learned is: Far too little time is spend on analyzing and designing the functionality of the business layer – domain models, UML diagrams, OR-Mappers, E/R diagrams etc. can all fail to describe the key components of a system, if you have not thought about business logic (even if it is just the simple “application logic” of which object can generate other objects, how links between “tables” is maintained and how security is enforced) and accordingly added functions and responsibilities to classes visible in your diagrams, and clearly named their responsibility in your code.

 

Allan K. Nielsen, Kindbergs Program Udvikling
Tweet This