Forced Domain and Forced Paradigm anti-patterns

These two common (and usually overlooked) “anti-patterns” are frequently followed by developers. Actually, I’m not sure if we can officially call them anti-patterns, but they are for sure common patterns that are not good to follow.

Let us explain what the symptoms are for each.

The Forced Domain occurs when a domain that is not the natural one for the solution is forced into that solution. Here forcing means using the foreign domain concepts and processes to solve the problem in a usually unnatural way. The most common example is the IT domain forced into business logic, user interface, user experience or any part of the solution that needs to be in the business domain. For instance, when we build software that presents data in tables, shows codes (data keys), exposes the notion of next record, the notion of indexes or even the notion of fields, we are forcing the database concepts to the client. To use the software, the user needs to learn those concepts first. The same happens for processes, when backups, initialization, remote calls and such are forced to the user.

Another case is when we are integrating applications. Each solution we integrate may have its own domain, and they may not be the same domain as the other application we are integrating with. Integrated applications should communicate in a lousily coupled manner. So, for one solution to use the other one, both will need to share some common business concepts and those should be only ones in between. For example, a banking system that offers loans may need geographical information about the land we are using as warrant. Forcing the banking system to understand about geo locations, azimuths and GPS readings is a non-sense. To obtain the needed information, the bank should request it using business information it manages, like a particular address. It is the job of the geographical system to convert that requirement data, in the form of a text containing an address, into world’s longitude and latitude pairs and convert the response’s information into readable data for the bank.

Ok, that may raise a question: Isn’t that forcing the bank domain information into the Geo-System? No, actually. The Geo-System is probably offering that query as a service, which means it is decoupled from the implementation, and using standard documents (which may even be generic) to pass info. Note that the Geo-System will not get bank specific information, just a street address.


Why is that Forced Domain a bad thing?
A domain is not just a set of concepts. It is also a very complex environment that has processes that are natural in the environment, working smoothly with the environment’s rules. The concepts fit and all data is structured in a way that facilitates the processing. If not, then the domain would not work. Think of a medical domain, in surgery, where the concepts do not match or where the rules make the surgeon juggle with tools not made for the job at hand. In the example, think of a surgeon that is giving a fish knife to work with, and that is forced to kill the fish before opening. The tasks of cleaning a fish would sound similar to some of those performed by a surgeon, but it is clear the final idea is different and that it is not a good idea to practice surgery with a fish cleaning knife.

Domain forcing also produces coupling of a very special kind. In some cases it will require application A to know about Application B to use it, and in some others it will force Application B to know about A in order for application A to use B! It is like teaching me mechanics so the mechanic can fix my car by asking me car construction questions. In collaboration contexts, our banking application is forced to store and process geographical data because it needs those values to query the geographical system.

The Forced paradigm anti-pattern is more related to development.
That is, when something is not done naturally as it should, but done my way. A very common example occurs when the developer, coding a services consumption client, uses an OO language. The service is usually defined as a port to which we send messages in a pattern. To do that in Java, for instance, we map the port and message idea to a method, defined in a stub, which acts like a local class. Thus, the sending of a message notion is lost. The developer sees the operation as just invoking a method locally. There, we are forcing one paradigm into another; we are forcing the method call operation into a message send operation. In Java we are supposed to have objects and invoke methods, not send messages. That is true, but then we should build an object that sends the message for us, and we just invoke that functionality. Sounds the same, but semantically it is not.

Is that bad too?
Of course it is. Forcing a paradigm causes impedance mismatch. It also increases overhead due to paradigm conversions, and removes semantics that would help the developer to create good performing code (like the stubs, may trick developers to use the call as a local one, when it is remote, and cause a bad performance). Actually, if you take a look at the current specifications for Java, we may see there are many that map the actual interface to objects following this antipattern, hiding away the actual process from developers. That is done to make it “easy” for them to use the API, but the resulting API would not be good enough.

Do you have any examples of these antipatterns? What is your take on how bad are they for your health?

Back to Posts! Blogging Season Ahead...

Ok, six months of not posting anything seems like too much.
There are lots of things that are keeping me busy. But I hope next months will allow me to finish the 4-5 posts that are just started but never finished.
I have one almost ready, but I feel I cannot hold it longer. So, it is free then to be published, although soon I may write about the topic once more, to clarify more things.

Off we blog!

Distributed, Integrated or Networked?

Distributed is a very common adjective. I usually tell my students to be careful when using it to refer to anything that gets in a network, which is usually the case. Even more, I tell them to beware of saying that REST or SOA are the answer to distributed needs, since it is not totally true.  The question I receive is then: When can we tell something is distributed or not? Ok, this post will try to clarify the difference between the three terms you can read in the title, as they are very often confused.

Let’s begin by thinking about systems. A system is a set of related components that perform some interaction, following some rules. The components may be anything, from hardware to software, for virtual to real, from people to things. Now, a solution is the implementation of processes and flows that yield the resolution of (or helps solving) a problem. The solution may be a monolithic thing, or it may be a system.

If you take a system, and then you analyze what is it doing, you may find that it may not be good to have all processing done at the same place. So, you decide to split it, and place some parts somewhere else. Think of tasks, all performed by one big process in one big environment. You may need to split the system and send some tasks to be performed in other environment, by another process. That way, you can obtain some benefits like scaling, encapsulation, “replaceability” and better use of assets. Some say you have also reusability, but that is not actually true all the time, as it depends on how you split the system, and that is definitely not the goal of splitting. But, you also lose some other things, like performance due to increased interprocess communication (IPC). Fair. You have there a distributed system, a whole that was split but still needs to work as a whole.

Now, imagine you have two systems. Each system solves a problem. You may discover that joining forces, the systems can solve higher level problems. So, you decide they must communicate somehow, to try working together. Fair. You then create communication channels and you have an integrated macro system. That is, each system is treated as a whole that may be composed, reused, to create a bigger system. We need to keep the individual system’s inner workings a secret, and treat is as a whole, so we need it encapsulated, reusable and loose coupled. We use standard interfaces, and we use higher level processes. Also, the frequency of interaction is low. And of course, the user knows this bigger system is a large composed thing, where we know the individual parts. We call an Integrated system.

But wait: isn’t that the same idea as above? Well, not really. It may sound the same, but when implementing, and semantically, they are not. See, usually distribution requires the system to work as a whole, and thus communication is a pain the user should not be aware of. It must be fast, and should be tight coupled. Since the parts of a distributed system are parts of a whole, they know each other and that allows for faster communication if that information is used in coupling.

Let’s check an example. For distributed systems, suggesting Services through messaging as the communication channel may not work, since that actually increases the overload for communications. (Distribution with services seems to be a bad idea, an antipattern).

On the other hand, you have the integrated system. Here each subsystem is a standalone system. There may not be the need to have fast communication, but reliable and loose coupled one. There, a service fits, since we don’t need to know the internals of one system to integrate it with another one.

Fine. What about the Networked system? Well, that is also a system, but made as a network. A networked system has its components interacting as a network. That is, you have nodes and communication between all of them, you have routing to get to a node, and you have a very dynamic node topology. Yes, you can have integrated systems and distributed systems running in a network, but that does not make any of those systems a network system. You may have the same parts of a distributed/integrated system in one big machine, no network, and still be integrated or distributed.

In a networked system, you can easily have more than one copy of a node. Routing and balancing gives you the ability to increase and decrease the node number and thus scaling in place, incrementally, focused. You can add more nodes without affecting the ones in the network, and you can create subsystems with them. In fact, all you can do with a computer network you can do with a networked system. I guess the difference between integrated/distributed systems and networked ones is clear. No? Ok.

Let’s do it again.
- A distributed system is a whole system that was split to improve its parts, that needs fast and reliable communication, just like if the component being called is near at hand. Usually communication is direct, tight coupled. It has a high interaction between parts. Finally, the reuse of part is optional.
- An integrated system is the one made of whole systems largely distributed, encapsulated. Low frequency of interaction between parts, loose coupled (high cost) communication. The parts are made reusable, and the user is usually aware of the composition.
- A networked system, all nodes form a network. An application is a composition of node services. Topology may vary, dynamically, nodes may not be standalone components, and it needs low coupling and other messaging services like routing and balancing. Nodes can be added and functionality adjusted without much impact. Scalable locally, and may contain subsystems.

Ok. One big networked system is the web. Now, in the web you can have integration (B2B transactions), distribution (an enterprise with subsidiaries) and other networking systems (maybe mashups, or 
Grids).

Clear now? Great! Based on that clarification, I pretend to write a future post: REST is tuned for networked systems! Stay tuned...

IWS 2010: The Why of REST expanded Abstract

Why do we talk about The Why of REST?.

I recall the old days, when I first met REST. It was introduced to me as a revolutionary way to create web services without all the complexity of SOAP and WSDL. It was an airline system used as an example. Very simple to understand. And yet very suspicious.

I had lots of questions in the following hours. The explanation was somehow light, and it seemed to me like a normal web request, nothing similar to a service. Later I read the REST dissertation (I actually did some annotations that I was to put in a blog post, and never did), and found out REST was an architectural matter, far more than just a web request.

I kept on the searching and found several examples claiming to be REST incarnations, but they were clearly not. I found several implementations were REST didn’t even fit.

How so? May you ask. Isn’t REST a solution for everything? Isn’t REST an API, or an API creation technique? Isn’t REST a Services creation technology? Well, no.

REST was not created to be used on SOA as a service creation technique. The origin of REST is quite different, and it is tuned for specific problems found when creating systems like the web.

Actually, REST is an architectural style, and it is made out of a set of different sub-styles, each of them is there for a reason.  Each style is a set of constrains, that has some benefits and some drawbacks, all carefully balanced to fulfill the need of a particular type of application.

Many tutorial show REST as something for the developer, when implementing a service on the web, full of tricks and tips on how to replicate the normal language and system operations over the HTTP protocol. Instead, REST should be seen at architectural level, where trade-offs are generated due to constrains, and for that we need to understand why each of those constrains are in REST to begin with.

This presentation will try to work on REST from the architectural point of view (Yes, no GET-POST-URI examples). Why is it a Networked System application so different, and why the REST constrains may help (after checking, of course, what those constrains are and how they impact the architecture). At the end, we will have some minutes to discuss in which SOA flavors does REST fit, and why is it not good for everyone. Several “whys” to explain, the big Why of REST.

What if you need Transactions and also REST?

Some time ago, in a discussion at REST-Discuss, en example for transactions in REST was given. The solutions are usually complex and tedious. This is part of one of my posts, fixed a little bit to include the sample case.

Case: We have a travel booking service. The traveler usually selects a flight and then a hotel to stay. If the person cannot confirm the flight, the room booking should not be confirmed. Here, we have a transaction. It is all or nothing. Good.

How do you model that in REST?

Usual method is to have two resource, the flight booking and the room booking. And, there is also another resource you create, that is a transaction resource. That transaction resource contains the transaction data, either what to do or what was done. Finally, you commit or confirm the transaction resource, and all is committed.

Well. For this, I assume there are two different data sources, and we can commit to each one separately, but when a transaction involves individual transactions at each source, then you use the famous two phase commit. Each source is a participant, right? The airline and the hotel are two different companies, so your transaction is not simple.

We can have one server dedicated to the management of transactions. We can create the transaction object in it and add the steps and then ask him to perform (it can even perform at each step and verify all worked at the end, or rollback at any step if needed). The problem is, given we need to send to it, manually, all the transaction steps and actions, that idead may suffer some scalability problems.

On the other hand you have the client that needs to do all that processing to commit the transaction. That is, the client needs to be transaction-aware.

My feeling is that exposing the data entities as resources, and leaving to the client all the commit processing, is exposing too much the application detail. May not break REST, but adds unnecessary complexity.

In the example, we model, for the two phase commit, two sources, the airline resource and the hotel room resource. It is them implied that both are like databases, even more, separated database engines. And, the client will have to drive the transaction management to change data in both and then to commit. That is implicitly forcing the concepts of a resource, but still it sounds like REST.

So far, so good. Now, my question would be: should I need to do all that to actually reserve a package using REST? Well, thinking about how would I do it, I'd actually follow an online reservation workflow and see what happens:

a. I enter and search for a flight. System returns a list of flights and I select one. At this time a draft reservation is created with my flight in it. (Think a PUT of the empty reservation followed by a POST of the flight).

b. Then the system offers me to add a hotel reservation, and from the provided list I select one too. That is added to my draft reservation (another POST).

c. Finally, I add my credit card information and post a confirmation (Another POST).

This last action is served by server number 5 of 10 currently serving. That server 5 needs to complete the POST, and if unable, it will return an error to the client. Well, that server uses the draft reservation resource information to call a transaction manager to commit all changes. If it fails, server 5 returns the error.

That is totally opaque to the client, which only confirms and receives a yes or no to that request. Depending on that response, the client retries, updates the selection of flights or hotel and confirms again, or even desists and eliminates the reservation. Simple, ha.

But wait, that draft sounds a lot like the transaction resource mentioned above. Well, yes, but semantically it is not. The user adding reservations to a draft is not expecting each step to be on firm. The actual transaction occurs at the end, when we confirm all the bookings.

The difference in this process is that client is freed from knowing the transaction is happening. Resources are just that, no databases nor tables that need transactions and the client doesn't have to choose the use of single or two phase commits. You can scale since you can change the number of servers or transaction managers without touching the client. AND, each client interaction leaves the system in a stable state. Actually, this can be RESTFull too!.

So, if we can hide the complexity of the transaction, why do we need to expose that complexity to the client? I may do it if that brings some benefit. My question will then be, which benefits will I found from one implementation to the other one, or why one of them is not suitable for some particular business case.

Look that the actual transaction coordination is happening under the cover, at server level. That data hiding allows not only flexibility to adjust the process, but also reliability, since failed clients can recover.

Cheers.

Trying to Explain App State for REST

I will try a different approach to explain the concept of application state, which is usually confusing and a recurring question from REST newbies.

I want to play blackjack. I have some friends and we all go to the online blackjack service.

Each one of us enters the application, which allows us to play BJ. The main screen requests our IDs. At the next minute, we all but one are ready to play (the one still not ready is because he forgot the ID number).

Let's see what happens there. We are, say, ten clients, each one is using the application, nine of which are ready to play while one is waiting for ID input. As you can see, there is one application, but each client has a different application state now. And each client knows its state clearly.

So, we left our friend finding his number and we started to request cards. We knew there was a dealer at the other end of the line, but we were not able to know if we were all playing with the same dealer!

Still, there was just one mass of cards. Yes, each card requested changed the mass. Still more, each request may be served by a different dealer!

Careful here. You may get confused. Requesting a card using GET is like asking the dealer to show you the face of the top card. You get a representation, but the card stays in the deck. In real game, you want the card sent to you. In REST, you will post something (a "draw a card" request message) to the dealer, with the actual deck URL. The dealer will modify the deck, say removing the top card, and will redirect you to that card. Here, the mass of cards is a resource, the dealers are the servers.

Now, can you clearly see the difference of the states? Each request will change the resource state, but that is not the app state. Each client has its own state of the game (each player knows its hand), and that is the app state.

The actual server state may be dealer working or dealer in a break. So server state is actually another very different thing.

So, what is the REST constrain here? Well, if you program the dealer to know the hand of each client, you are violating REST, since then the server is actually controlling the state of the app for each user.

The idea is the client controls its app state, not the server. So, even when the client wins the hand, it will have to show it to the dealer, saying I won. And the dealer will give me the lollipop gift. And after that the dealer will wait again for another request.

Each request will contain all the information needed to be totally completed in one interaction. The sum of all interactions is your final goal, the application.

Also note the resource can be changed in the process, but that is not an app change.

Why is all that setup good? Well, you can have thousands of clients that it will not impact servers, since they do not keep track of the hands. You can add or remove dealers with no impact on the clients nor in the application states. You keep hidden the resource implementation. And you got then a nicely distributed game, scalable and simple.

Hope this helps!

SOA discussion remembrances

Back on the year of 2007, end of May, there was a post on The Server Side by Albert Mavashev that highlighted the growing complexity of SOA, why architects should take that into account, and stating that a balanced, incremental adoption of SOA was one good idea. Problem was his starting paragraph, where he says:

SOA (Service-Oriented Architecture) seems to be the buzzword everywhere!!! SOA will solve all the problems, everyone wants SOA -- or so I hear. Anyway, I remember similar claims when CORBA was coming to town. It was supposed to change the way IT does business. To me SOA is XML/SOAP based CORBA with evolving standards around it... I might be wrong.

That was a very long post, in terms of comments. The CORBA comparison acted as a flame bite, and we ended with 169 comments or so, of which 20 were mine!. At that time I didn't even have a blog, so this post contained the first discussions and comments representing the first attempts I did to expose my understanding about SOA. Reading them back today, I feel I can adjust some things, but must of the things remain the same, and I feel that when I wrote all that I was just a kid!.

I wanted to rescue my comments, what effect they had, and comment upon them in this post. Old memories!

The discussion was started already, with lots of enthusiastic people and some angry ones too. There was a comment line that included James Watson and Rodrigo Cana. James posted a link to a Thomas Erl article about SOA . The discussion was around if SOA, CORBA and EJB where the same thing under a different disguise. Rodrigo was saying:

I read the article, and I've never read that much bullshit about SOA.
Really the author does not have a clue about what SOA is about.

But I was relieved to read James Watson's post, that tried to clarify that SOA is not about tecnology to integrate applications. Instead, it is about how to manage the integration in a way to ease manageability and increase flexibility. So, if your "SOA" usage is increasing complexity, then you're applying the wrong concepts.

I read the article too. It seemed nice, but I felt that some misconceptions were given by granted, just because they were the current way of doing things. As a defender of concepts and theory against the mainstream, I answered to Rodrigo.

Rodrigo.
I'll jump in this line since I feel is the most sane in the whole discussion.

Yep, I read the article too, and although the guy has very good lines in it, it starts by falling into the same error a bunch on writers here
falls too. Just in reading the part of the vendor-oriented service-orientation, Thomas mentions something like "core expectation of SOA is its ability to harmonize and streamline diverse technical environments". Probably it may be a sell point for IT, but in fact thatis a side effect (not often achieved). The core expectation in an SOA is to model an application in the problem space using a business service metaphor.

In another line, Thomas says "Like traditional multitiered architectures, SOA is based on a model wherein solution logic is distributed". That is a trap. Services should be thought as complete business functionality offered no by methods or classes, but by services. Distribution leads to segmentation of code and business logic, and it is a technical strategy. SOA is about definition of business functionality as a whole (which may be internally implemented as a distributed thing). That is a bad approach, SOA is not to distribute, it is not DCOM over HTTP.

I Agree with Watson too. Although, he is still thinking in SOA as a technology issue pain reliever. SOA is about managing not technology but business. And the complexity is not about the concepts or technologies used, is about the impedance mismatch obtained by mixing them.

Agree, again, SOA is not a silver bullet. Jump into the buzz word bandwagon antipattern

Ok, I think the part of the expectation has changed a little bit for me.
Actually, I feel SOA is more in the integration line than it is in the distribution one, and I will blog about it some day. Back then, I got SOA more on business than as a technology for integration, I still think that.

James was back saying I might treating Erl in a way he didn't deserve, and actually I granted he was not suggesting but explaining how things were done. So, sorry Thomas.

Later, I kept reading another branch, where there was much discussion about the Hype and about CORBA as a SOA sibling. I answered a comment by the author post, Albert, saying:

Hi Albert.

Hard to keep on reading all the messages (jumped late to the discussion). But I think I have a couple of interesting points I want to highlight.

1. Goosh. The debate between WS people and Corba people is always the same. I do teach Web Services course at university, and I see the same ideas back and forth in my students. WS IS NOT distributed components.
CORBA is about a broker pattern for distributed components, and WS is about business service metaphors and how to communicate with them using messages. Apples and Oranges.

2. SOA refers not to a technology, as many pointed out, but to a supra design in the problem space. Yep, nothing related to the implementation, but to the business level. Services are aimed to architecturally designing the business processes using the Service metaphor as the building blocks. The problem here is designers are not architects nor business people, but plain developers that, by the way, worship OO and RPC.

3. Getting to your point, I understand CORBA suffered from complexity in implementation, and lack of governance tools. But Orchestration (different from governance and even choreography) probably was not needed (there was the broker in the center). Services are other things.
The messaging system is the one that requires orchestration, choreography and governance. Services need to be designed, as business entities, with lax interfaces that are business oriented. No RPC. And so, you need also governance of business processes. Since the actual market sells tools to OO developers, they are oriented to the RPC governance and emphasizing the call flow and not the messaging manipulation.

By this I mean that the tools should have two levels, the one for developers, that should be in the cubicles implementing the utility classes for the service implementation, and master designers with business background designing the services per se. And then, IT support controlling the messaging system pitfalls, and business people controlling the business flows.

If you want to compare CORBA and SOA complexity in governance and orchestration, then I would say they are different. And I would say lots of people don't really know what are they calling SOA anyway, which makes it even harder.

On another branch, Jose Romero and James Watson were discussing loose coupling (the original branch idea was that CORBA was not the same as SOA, and later that SOA was a hype worse than Web 2.0). Well, Jose was mentioning a relation between coupling and synchrony:

In response to james, there are 2 categories of coupling : (data) type coupling and temporal coupling. Type coupling is the most obvious : the type of exchanged data induce coupling between client and service. Using literal xml document preserve the structural semantic and allow for flexibility/variability without adverse effects (in fact infoset would be enough to convey data). Temporal coupling is more subtle. To support message exchange pattern other than traditional request/reply service must support asynchronicity. It doesn't imply that the message exchange pattern must always be asynchronous. To reduce temporal coupling between client and service allow for a more robust system : partial and transient failures don't block the whole system

I jumped in:

Jose.
I would thing there are more that only 2 types of coupling. Actually, coupling is a generic thing that feeds from several variables.
The "synchronicity" you mention as temporal is one that is not as trivial to spot. Data typing is another. RPC "exchange" pattern (I hate to call that), is another one, since it forces a semantic in the message body, even a pre-stated structure of operation and parameters with a return value. The transport may be another one, and there is where HTTP nature, for instance, is confused with "synchonicity".

How come? HTTP is not a messaging transport. It is a Hyper Text transport, based on the REST idea of distributed resources and local states (although REST came later). No messaging anywhere. Thus, since HTTP operations require a response, and that means they need to be a blocking pattern, doesn't mean that the messaging that piggybacks on it should also be. As Hohpe mentioned in a conference, there is no message that goes and comes back. There are actually two messages. The fact that HTTP is as good to send a message in a request and bring the other message in the response, shouldn't tie the blocking or synchronous pattern to messaging as well. IN other words, you can have the request response without needing a synchronous call.

Lastly, and that is something I always say to my students when talking about products, I see no client-server relation in SOA messaging. I see an origin of the message and a destiny. That final target contains a service that consumes the message. A service is not method exposed. It
is a cloud of coherent functionality that is activated by messages, and that may send a message back to you.

I love that vision

Then, Constance Eustace opens a new branch ranting about all the guys that just say SOA is not this and that, but no one states what it actually is. Excerpts from this:

"SOA is distributed procedural programming. Services are stateless request-response calls on arbitrary destinations. "

"SOA seems to be going after a high-level design of simple procedural calls and data retrieval/updates/deletes/inserts, with a "unified" calling interface, if one can call the current state of service/webservice acronyms unified or simple."

"SOA is an attempt at simplicity that has failed:
- technologically

• too many different standards
• no agreement as to what the hell it is
• overly complicated (failing its primary mission: simplicity)"

To this, I obviously answered. In my response I stated a couple of things that may not hold totally correct. For instance, I say services are stateless, and they may be but that is not actually a requirement. I also said, SOA was an architectural Style, still there is no definition for it. I will try to defined later in another post. Here is my answer:

I agree with several things you say. I disagree with the overall idea.

Yes, as the hitting the moles, each time someone tries to define SOA (a mole getting out of the hole) someone crashes a pole in the head of the poor thing just because it didn't understand SOA.

There is a difference between what SOA is and what SOA has become due to development incompetence.

SOA was confused with distributed procedural (or object oriented too!) programming.

Services are stateless, but they were forced to be RPC calls.

SOA was forced to sell the idea of a simpler, unified, platform independent interface, which is not (a service IS NOT an interface).

SOA was sold out as simplicity when it was just a way to architecture using the service metaphor.

SOA has a different vision, not huge, not grande, not even new.

CORBA is based on an architectural pattern (also called Style) that is the very well know Broker pattern, that was sold out as an implementation (the same all vendors try to do with SOA).

SOA is an architecture style, where you model using services that are named functionalities accessible by messages. Is not panacea, and is not simple. That is why architects (and I mean Enterprise Architects) are the ones that should deal with it, and not Java programmers.

Finally, simplicity has nothing to do here. Trying to do SOA without thinking in architecture and with the actual tools, creates stovepipes systems, impedance mismatch, little monsters that make overly complex to call a method using a service metaphor with asynchronous messages.
Creates chaos.

The question would not be what is SOA and what is not. The question should be Why are all these people trying to make the square SOA vision fit into a round hole? It is hurting a very good abstraction that, by the way, is not good for everyone.

Finally, we would never agree on what it is, since every kind of "hammer" looks at it as its very own kind of nail.

Following that rant, Jordan Xu justified using an art analogy that I found amusing. He said:

"They want to take IT into the realm of abastract art. Ever feel like being in an art museum, standing in front of huge canvas with a big red square in the middle?

Anyway, they can't justify their "Chief Architect" title without an "architecture" to preach for. Since they can't talk about anything concrete and get caught with their empty rhetorics, they hype up something abstract like SOA. No one can say they don't know what they are talking about because nobody knows what it is"

Interesting. What Jordan describes the architect's do is an antipattern called Intelectual Violence. He may be right for some architects I've heard of. Still, I answered to the point that SOA was an abstract thing, and actually to the abstraction concept:

Well, Jordan, at least we architects sometimes do agree about something.

This is not about who is right and who isn't. Is not about art (abstraction is the elimination of details to model a reality, not Picasso's paintings).

By the way, abstraction eliminates details but keeps structure, which makes what someone says a very concrete solution.

I have a hard time trying to make developers crack their shells and see what other people are doing. I'm being unsuccessful to bring a Hibernate user to think abut XML databases, or even Object databases. There are lots of java developers I check their code and see procedural programming, no OO design anywhere. The same is happening with services (forget SOA). Even other people teaching at the university see Services as objects exposing their methods, and there is no way to change that mind. If they cannot understand services unless I talk to them in terms of objects, then they will never understand SOA.

At least, my new hope are my students. Some of them are starting to investigate a cross model representation. Actually it is hard, they say, and the tools are not of use, but at least they have the concept pure.

By the way, I'm a hard core developer, differenec is I swallowed the right pill.

There is actually a paradigm shift that is needed for modeling and implementing. DOSE for instance (Document Oriented Service Engine) is still in napkins drafts, but the idea is not to use stubs for web services, but message brokers instead.

Then Jordan came with an interesting take on interfaces that started another discussion flow. He said:

"The concept of breaking software systems into independent, reusable service components has a major flaw. It does not address the changing needs of the business environment. Many of these services, such as "process payment", may be considered permenant, but the devil is always in the details. What are the input and output of these services change all the time. By fixing the interface to the services, we often made them very inflexible. If even the reusability of plain objects are often difficult, how can we be so sure that we can define services that are constant(in all details) and thus reusable?"

Reusability word was there. I know everybody things on services as reusable things. I have had some discussion about that in other forums, and the idea actually is tracked back to when objects ruled the earth, and thus my peers said services were an evolution from objects! Still, I don't like the reusability thing. I feel you should create services to be composable, not reusable. I posted this:

I see a repeating idea about SOA, services in particular, that doesn't fit in what I have learned from my childhood as an architect. I will explain myself.

I recall the old Object oriented days when all the fuzz was about reusability. The idea was clear: separate the code into coherent and functional complete things. Then a problem was found: the interface. You could not reuse the thing because of the interface not being suitable, or requiring things not in the new problem domain.

Then a service came. It was the idea of having a business functionality (not an interface, not an object) able to accept messages and offer a business value. It has no reusability concept, the fuzz was actually about replacing fixed interfaces with a simple port. And somebody said:
Let's implement it with SOAP. And SOAP was Simple Object Access Protocol, an RPC oriented wrapping protocol used to access distributed objects. Wow. Wait a minute. Something is not working here. I know objects were supposed to accept messages. Nobody uses that. But a service is not an object. A service has no methods or parameters. Definitively, it has no response. There is no interface per se, just a port where you deliver your message.

Now, the message is not SOAP. SOAP is a wrapper. The real thing is in the body.

Once the service has read from the port the message content, it will now know what to do. And it may decide to send another message. As simple as that. But as difficult to grasp for an object programmer, that it was reduced to a YAMTCARO (Yet another method to call a remote object).

NOw, can somebody tell me why a service needs a method name to be called? SOAP had stubs in the client, to simulate the remote object as local. Services are not object, stubs have no place here. Makes no sense. You need a port object to stream your message. Simple.

Finally, since a service is just a port, and the service semantics (which are not technically describable) is hidden behind the service cloud. You can create a versioning system underneath the cloud, so old messages can still be accepted and new messages, that represent new business needs, be processed as required. Service cloud still as white as before. It may have a new port, but old ports may be still there.

Enterprise architects? Their job is to have one eye in the problem space, and the other in the solution space. They will work with business people to design services scope (business driven), and work with designers to technically create the structure of the cloud.

Hope this helps to understand why I say all that I say.

The concept of services with a uniform interface is also base at REST architectural style. There is the idea of that being impossible? REST is a working thing, and that interface is possible, sure it is. Well, Jordan told me that concept was far away from mainstream understanding of SOA. Made me feel as the only one with that idea:

"William, your vision or interpretation of services in SOA is very different from the mainstream. The most popular description of services is a set of independent, reusable functional units that can be "reassembled" to form different applications. I am not sure how many architects will agree with you."

Oh, well, time will tell.

Meanwhile, Karl Banke took the idea of the port and managed to create an analogy to RPC. Very interesting use of a metaphor (I saw it later again, mentioned by Guido):

"Well, if a service is just a port, there must be some semantics for the port to know what to do with the message is receives. While there might be multi purpose messages, they are even harder to maintain than single purpose messages. So, almost all messages to a port will essentially be "single purpose", like "make a booking", "transfer money", "get account statement" etc. Lets call this the "function name" or "message type". Of course the client will usually have some kind of return message, which, for the purpose of this discussion, shall be called the "return value". The inner content of the message shall be called the "argument". What is so different from a method call after all. Essentially nothing. For the sake of simplicity, we might even have some fun, and create an EJB or Corba Object, that we call the Service Port.

What you are doing is "de-abstracting" away from perfectly valid abstractions, arguing about the service cloud that might have a silver lining."

Interesting vision. My students did ask the same question. At the end, sending anything to a port can be mapped to an RPC call model. Humm, I'm convinced that is not the case. Actually, document passing has not more semantics for the port than to receive a document and deliver it to the appropriate reader. And sending back a response is totally up to the reader, by the way. So, I tried to answer that with an example:

Service is not just a port. A Service is very complex thing that has a simple mission: to provide a business functionality as a service.

How do you communicate with it? Using a port. You send a message to a port. That the message body contains an XML structure that is thighly coupled to reassemble a method call, is not of the Service business, it just doesn't care. That is, using that inflexible analogy makes your service miss those important features like being weak coupled, but it is your call.

Now, what separates an architect from a designer/developer is that the first one works with the service image I told you, business. The second ones will take that vision, see how can they publish that port, what do they need in the message or in the port itself to determine functionality, create a message processor that will route the requests to business logic and other levels, and then finish the work. The easiest way is the lazy programmer way. They put all the business logic in one class, create a method to publish it, create an XML schema that reassembles the method call, and use the port to get that XML. In other words, they are downsizing the metaphor to an RPC that may be far less expensive if done with EJBs or even CORBA.

In other words, it is valid if you want to make your service look like an exposed object. I see no point in it, though.

Now on your examples: "make a booking", "transfer money", "get account statement". I see them clear as fine grained method calls. The first thing you hear in the street is to get away from those. They are expensive. Your need to create course grained services. The next example may seem a little extreme.

Service: Loan service.
Functionality: Actually, the service is able to process loan requests, payments, notifies due loans and you can ask for a loan status.
How is it done?: There are four document types, each one has a particular schema: LoanRequest (input), LoanInfoRequest(input/output), LoanNotification (output) and LoanPaymentNote(input). Any input XML document may be sent to a port located in www.willyloans.com/loanport or to the email loanport at willyloans.com. There, an XML processor will determine the schema type, and send the actual data to the loan analysis department, to the loan payments subsystem, or to the loan service desk system. You can guess what each system does, and what each document may contain.
You can create four WSDL if you want. All pointing to the same address. Just make them document style.
More functionality? Just add it. Another schema and that's it.
Versioning? No problem, several XML databases already support that.

What do you think?

That answer drove another set of comments about scalability using strings and James Watson saying it makes no sense to talk SOA in technology terms because SOA is about business. I couldn't agree more with that. Still, I tried to answer all those topics:

1. A WS with a string parameter may fit for a very tight concept of service. You send a document, and a document does not contain just a single data, but a structured or semi-structure sets of information. You can compress that into a string field, no discussion, but that I wouldn't do.

2. Then, a document is not just a couple of parameters. eXML standards and industry are creating XML documents that are in the order of several megs. Pack that into a string. Parse from XML to String, then encode it to add it to XML SOAP, then decode it, and un-parse it. Make 100 concurrent users try to send the same document to the service. You will receive less that 1 tps. That is a scalability problem. And since RPC is a blocking mechanism by nature, the problem is reflected in the clients too! I have several stories about clients timed out.

3. Still not sure Services are stateless. I know communication between them is. A service is not only a process, it is functionality and may have a state, although they are implemented using a stateless object (EJB).

4. Guido is totally right, Services scale by concept, implementations draw away the scalability.

Still, John Brand was interested in my product comparison. He said that going from 1 to 100 users sending one document decreasing the TPS was not scalability problem but a performance one. Also, he did not see the impact on the architectural perspective. Here are my ideas on that:

1. Probably the example was extreme and confusing. Performance problem is when your "expected normal" load is returning a low TPS. Scalability is when you want to increase the load, using more users or bigger documents for instance, and the expected growth lowers dramatically the TPS. So you are right, the example was missing the increase factor. I assumed 1 user and then grow to 100.

2. It depends on what is "architecturally significant". From an architect point of view, I may want to use a service oriented interoperation between two of my partner companies. If I know both companies work with EDI formatted documents, and I see that my architecture design will work only if those are kept in place, I may decide to dig to that level and indicate the use of EDI payload instead of XML. To me, that is "significant" to deep my hand on. As such, I may also indicate to designers that we must use message queues instead of XML-RPC, for instance, since we are dealing with As/400 machines and legacy code.

3. About the scalability, that is a quality property that architects usually look at. I know that the service I'm asking to create is a very sensitive one. And although I have just a few clients now that will use it, I foresee in a month I will have 5 times the clients, so I need it to scale. That is "architecturally significant" too.

4. Finally, I may want to handle the spec, or simply tell to the designer: I need it to scale to 100 users and 10 meg messages. I wrote an article about the need of messages with a full, huge payload. The idea is actually to separate the bulk data from the message, what you don't have to do is create messages with bulk data. message are not good carriers of huge payloads. Use raw xml over html or keep the payload in some place, but not to "call" the service.

Further down, I found Paul Beckford. Paul is a very nice guy to discuss with, and I like very much to discuss and drinking good coffee. He asked a very small question:

"Are you talking REST?"

Ha! My Service loan example reassembled much of a REST implementation.
Interesting. Still, back then I was convinced the web services built using REST were not web services, where just RPC in the URL. Now that vision has changed, because I have seen the correct way of doing services using REST constrains. Still, that comment was a summary of my REST position:

Well, Paul, I'll answer that one.

REST is another word several people use, but may not be clear of what it refers too. Chapter 5 of Fielding's dissertation explains REST. It is an architectural style used to create distributed hypermedia systems.
Doesn't sound like services, right? Because it was not created for them.

REST is based on a state. Imagine resources in the net. The resource as a state that may be changed by simply performing a reduced set of operations (post, get, put and delete). The resource is accessible using its ID (URI). And you work with a representation of the resource.

For most people, REST-like web services is simply calling "services" using arbitrary XML documents and http. It's not that easy. There are lots of confusing ideas there. For instance, there are people saying services are stateless. But resources is REST are not. The communication should be stateless. Confusing, right?

Web Services Architecture spec has a view of the architecture where the services are seen as resources. So, we may even say that Web Services using SOAP are actually REST-Like!.

REST has not idea of messages. And it is for hypermedia. That means that web services using message queues or some other transports have no meaning in REST-world.

Services as resources. Services do not have a representation, they have a port that is actually the web resource used.

When you request a page in the Web, the representation of that page (the HTML code) is self-sufficient. The browser will paint it without trouble. But with an XML document, you as a developer must know what those tags mean to actually do something, it is not self-describing as REST request.

For this, and for several other dicrepancies, I thing that REST style web services shouldn't have that name, since they are a missreading of the REST idea. Confusing, right?

Well, Paul, I think the short answer is that the above ideas of services, (extracted from the spec, not the vendors) are actually implementable using document style web services, (SOAP and WSDL), and also using what has been call REST-like services. We can even implement them using message queues, SMTP, or plain TCP/IP sockets.

Funny, huh?.

This may go to my blog someday

Paul answered some very nice things. I understand I usually go too academic, since I'm of the strict definition, correctness driven type of guy. Still I understand practice is not always as theory dictates. Just like the quote I found in Nicolai M. Josuttits' SOA in Practice Book, that he reproduced from Laurence Peter "Yogi" Berra:

"In Theory, theory and practice are the same. In practice, they are not.".

Paul ended his comment with this:

"I agree that many do find REST confusing, but I don't believe the underlying idea is confusing in itself. It is one of those ideas where less is more IMO. I find that people (including myself) struggle to come to terms with the less.

The central concept behind REST is the idea of a uniform connector, which seems to fit your "document style" "distributed architecture" idea pretty well (I am avoiding the use of the term "web services" since I believe it as been polluted in much the same way as SOA). As for SOAP, there is very little uniformity there. Hence I agree you could implement REST with SOAP, but to do so would miss the point.

Anyway, thanks for a rare ray of intellectualism on TSS. I have followed this discussion and felt no need to participate up until now. I will continue the discussion on your blog if you don't mind.

It is nice to meet someone with something thoughtful to say".

And to my blog he went, with very richful discussions!.

But the perils didn't stop there. Guido Anzuoni stated, with much of reasoned logic, that all went down to XML again. After that, many other comments started talking about implementation. Even lokesh Pant cherished the change from 10000 feet view to implementation one. All was interesting. Then I found Karl Banke dissecting my response and commenting about what I was saying. He thought I was "seriously on the wrong track", at least about my way of looking at services as business functionality units. Actually several people told me that, but I feel relieved to learn there were other not thinking like that, like Nicolai.

Well, I answered Karl, again:

"Ok, Karl.
I think you an I agree on several things. I have a problem sometimes when people doesn't clearly understand what I'm referring to. Let's see:

1. A business functionality cannot be abstracted in that way. There is a wrong concept between developers, that assume abstraction is talking about business needs, opposite to implementation. Abstraction is creating a model of something, with less detail. You can abstract your implementation using a model of classes in UML, or you can abstract a business process using a business model. Problem comes when developers try to mix the two.

2. A service offers some functionality. That should be clear. The developers implements all he needs for the service to work. Simple.

3. "Object perform(Object)", as I say, it's your call. That assumes objects in the platforms, assumes a response, and assumes parameters. That is coupled to me, you only "loose" the particular object type.

4. My notion of an architect is much more complex. I teach at the university that course. An architect looses some detail as a master designer, to obtain it back as a business analyst. As I said, Architect is not the oldest programmer in the company, it has competence in six major roles or subdomains. But I cannot comprise six moths of lecture in a paragraph.

5. Far away from what I have said in this thread. Services are not for everyone, they are not panacea nor silver bullets, they need more an architect than a java developer, and they won't be useful or even different from what we already have until the mainstream stops buying the vendors interpretation of a YAMTCARO, and requirement analyst stop asking the developers to create services just because everyone is doing that.

6. As you see it, it may be right. "Make a Booking" seems to me a perfect task for a Booking Service. You can also add a another task to cancel a reservation and even request an availability report. All of them perfect tasks for a Booking Service. Those are called "operations".

7. Lastly, I mention a "complex thing" not because services are difficult to create. They are just not as simple as creating a method with a string parameter.

Hope this helps."

And. OOOPS, it started a discussion between Paul, Karl and Guido. Paul was telling he thinks all my ideas were actually REST, and that I was confused, and posted some examples of why with REST the universal connector (as he called it) made interfaces that didn't change, as opposed to WS-*. Karl was saying we are were discussing bizarre things since technology has nothing to do with never changing interfaces. And Guido stated the uniform connector added just another layer of abstraction, but the coupling of was still there since callers and providers need to agree on a contract anyway.

Well, I guess all that sounds interestingly correct. But there are some missing points in there that made them correct and missing the point at the same time (hummm). For instance, in REST you discover the flow from links in the hypermedia documents. The same with WSDL, problem is people uses WSDL as a static thing, design time. Karl was correct, but REST is not a technology per se, ha, and some technologies do force changing interfaces. Of course we need to agree on something! Guido is correct, but the coupling is not in in the interface, but at the business level, where we want it to be.

Ok, the comments went down to discussing what a uniform connector was and why CORBA IDL was not for the job, and what benefit did that bring to the implementation (to have a uniform connector, that is). It is clear in the REST dissertation, though.

After a lot of comments discussing who was right and actually saying the same thing over and over again, I said good bye to Paul stating that Uniform Connector was all that I mentioned about the port and just sending messages to it.

Yep, it is a good thing to remember old days.