Architect's Thoughts

Dealing with versions in REST? Think again, you may not need to.

2011-05-01T17:09:00.002-06:00

In the recent IWS 2010, I saw Zachman presenting an interesting note about enterprise architecture. He quoted Brooks by saying that redundancy drives the system to chaos, entropy. Well, actually some “managed” or better said “controlled” redundancy is necessary to allow scalability. In a nice context, that redundancy is a “mirrored” one, a copy, an instance. That means each part that is redundant is the exact same copy of all other redundant parts. But not all is so nice: there is a meaner redundancy, the one created by specialization or customization. That one is partial redundancy, where one part of the instance is the same for all instance, but there is a part that is unique, different from the others. When that happens, we usually say we have a new version.

So, version is a way to differentiate between two things that are similar but not quite the same. Version is something like an ID. That is partial redundancy. If there were no redundancies, or if all redundancies were absolute, then we would not need versions.

Now, if two versions of something were to be treated the same, no difference in processing them, then there is no need to differentiate. In other words, nobody cares about the part that is different in the instances. Thus, version can be defined as the ID plus the special processing or treatment that ID requires.

Let’s look it using another perspective. To create a version, we need to change something. That is: data, structure, a process, something that is changed. The original form is kept and the new form is assigned a version ID to differentiate it from the original. In other words, versioning is basically a technique to handle the differences that appear when something changes. The real issue is then modifiability.

Modifiability? When we modify something and the rest of the system has to adapt to that change, we say we have no modifiability: the system is tightly coupled. Modifiability is a quality property that allows some things (even instances) to change without implying the system will break. It is the capacity of the system to support the change of one of its parts (here, each individual instance is counted as one part). When something changes and we start creating special version of things to handle the change and keep the old parts still working, we are adding partial redundancy. And that is bad, remember?

So, as a piece of advice, try to stay away from versioning as hard as you can.

But, what if you need to use versions, how do you work with them in REST? Well, REST is a style that welcomes redundancy, but tries to keep the flexibility to change without disrupting the system. That is, it offers modifiability.

Let’s see what can change. In REST we have different servers that can come and go. Topology can change. Since we are using layers, the fact that a server changes in a second level or beyond does not affect us. If the server that goes away is the one that we are talking to, there should be another that takes its place (redundancy). If there is no other, then we are in trouble. So, here we have total redundancy, not partial.

Now, let see the next level. The resources that are actually operated by a server may change. Say, we have no longer a text file but a database, or the resource is no longer a static image but a video, any change that a resource can suffer. Since we manage resources through representations, that change may not be a problem. It is solved by keeping the same representations as before and even adding new ones taking advantage of the changes in the resource.

Ok, we may say: what if the representation is the that changes!? Well, here we have two types of possible changes: Structural or content. If the representation structure changes, then we may be in front of a totally new structure that needs a new media type. Maybe the structure changes a little, like in WSDL 1.1 and 2.0. Well, in that case, the new 2.0 needs to get a new IANA registry (actually it has an application/wsdl+xml assigned for 2.0) . The client will know what is being sent by reading the media type. Humm, ok, it may not happen, new types for new versions of already existing media types may be difficult, in that case it could be the new type description contains backward compatibility, or simple that the old spec is not supported anymore.

Now, let’s analyze content modification in the representation. What does it mean? I guess that the usual data that was being sent by the application is not the same as usual. Humm. That means the client is statically bounded to a set of values? Well, that is not a problem of REST, right? The content should then have a way to tell the client what has changed. A version you said, well yes, but application driven. REST has no restriction of what values or fields you sent out, one of them could be a version number, but that is the application data, application semantics, totally out of REST concern.
A little note here. We may have a well-known generic data type like XML. And we may have one normal document structure the client is reading. What if we add some new elements with some new data, and remove others? Well, here we are not changing the data type (XML is still XML). What we changed is the XML schema. And of course, the schema is specified in the XML body. So, a client can perfectly parse the XML if it understands the schema. Again, nothing related to REST.

Let’s continue. Now, I want to change the actual process. Say, we need to do some posts to complete a process. Suddenly, a new Post and a verification Get is required. In this case, unless the client is a static thing that is not following Hypermedia as the Engine of Application State constrain, that is solve alone by adding a couple of links to one particular resource representation.

Ok, the resource that is changed with two more links may be the one to be versioned, right? Well no. If you still need the same resource representation, either create a new representation, create a modified special representation based on the client request (may be a query parameter) or create a new resource that will be used upon a client selection. Think of a virtual store that has a normal four step process to check out. If you have a credit card registered, you may want to skip the credit card form, or even perform a one click check out. Note that in these cases, we are simple adding options that will drive the client to new process flow, old clients (that only understand the full checkout process, although that is not nice implemented) may be able to follow that old process link. So, we keep all old thing add add new ones without breaking the app, the magic of Hypermedia.

Did I forget something? I guess not. All the cases can be resolved without making artificial URLS with versions or creating custom media types. The only cases where a version id may be needed are the ones that are application specific and not related to REST.

Cheers!

Forced Domain and Forced Paradigm anti-patterns

2011-04-21T14:54:00.000-06:00

These two common (and usually overlooked) “anti-patterns” are frequently followed by developers. Actually, I’m not sure if we can officially call them anti-patterns, but they are for sure common patterns that are not good to follow.

Let us explain what the symptoms are for each.

The Forced Domain occurs when a domain that is not the natural one for the solution is forced into that solution. Here forcing means using the foreign domain concepts and processes to solve the problem in a usually unnatural way. The most common example is the IT domain forced into business logic, user interface, user experience or any part of the solution that needs to be in the business domain. For instance, when we build software that presents data in tables, shows codes (data keys), exposes the notion of next record, the notion of indexes or even the notion of fields, we are forcing the database concepts to the client. To use the software, the user needs to learn those concepts first. The same happens for processes, when backups, initialization, remote calls and such are forced to the user.

Another case is when we are integrating applications. Each solution we integrate may have its own domain, and they may not be the same domain as the other application we are integrating with. Integrated applications should communicate in a lousily coupled manner. So, for one solution to use the other one, both will need to share some common business concepts and those should be only ones in between. For example, a banking system that offers loans may need geographical information about the land we are using as warrant. Forcing the banking system to understand about geo locations, azimuths and GPS readings is a non-sense. To obtain the needed information, the bank should request it using business information it manages, like a particular address. It is the job of the geographical system to convert that requirement data, in the form of a text containing an address, into world’s longitude and latitude pairs and convert the response’s information into readable data for the bank.

Ok, that may raise a question: Isn’t that forcing the bank domain information into the Geo-System? No, actually. The Geo-System is probably offering that query as a service, which means it is decoupled from the implementation, and using standard documents (which may even be generic) to pass info. Note that the Geo-System will not get bank specific information, just a street address.

Why is that Forced Domain a bad thing?
A domain is not just a set of concepts. It is also a very complex environment that has processes that are natural in the environment, working smoothly with the environment’s rules. The concepts fit and all data is structured in a way that facilitates the processing. If not, then the domain would not work. Think of a medical domain, in surgery, where the concepts do not match or where the rules make the surgeon juggle with tools not made for the job at hand. In the example, think of a surgeon that is giving a fish knife to work with, and that is forced to kill the fish before opening. The tasks of cleaning a fish would sound similar to some of those performed by a surgeon, but it is clear the final idea is different and that it is not a good idea to practice surgery with a fish cleaning knife.

Domain forcing also produces coupling of a very special kind. In some cases it will require application A to know about Application B to use it, and in some others it will force Application B to know about A in order for application A to use B! It is like teaching me mechanics so the mechanic can fix my car by asking me car construction questions. In collaboration contexts, our banking application is forced to store and process geographical data because it needs those values to query the geographical system.

The Forced paradigm anti-pattern is more related to development.
That is, when something is not done naturally as it should, but done my way. A very common example occurs when the developer, coding a services consumption client, uses an OO language. The service is usually defined as a port to which we send messages in a pattern. To do that in Java, for instance, we map the port and message idea to a method, defined in a stub, which acts like a local class. Thus, the sending of a message notion is lost. The developer sees the operation as just invoking a method locally. There, we are forcing one paradigm into another; we are forcing the method call operation into a message send operation. In Java we are supposed to have objects and invoke methods, not send messages. That is true, but then we should build an object that sends the message for us, and we just invoke that functionality. Sounds the same, but semantically it is not.

Is that bad too?
Of course it is. Forcing a paradigm causes impedance mismatch. It also increases overhead due to paradigm conversions, and removes semantics that would help the developer to create good performing code (like the stubs, may trick developers to use the call as a local one, when it is remote, and cause a bad performance). Actually, if you take a look at the current specifications for Java, we may see there are many that map the actual interface to objects following this antipattern, hiding away the actual process from developers. That is done to make it “easy” for them to use the API, but the resulting API would not be good enough.

Do you have any examples of these antipatterns? What is your take on how bad are they for your health?

Back to Posts! Blogging Season Ahead...

2011-04-21T14:52:00.000-06:00

Ok, six months of not posting anything seems like too much.
There are lots of things that are keeping me busy. But I hope next months will allow me to finish the 4-5 posts that are just started but never finished.
I have one almost ready, but I feel I cannot hold it longer. So, it is free then to be published, although soon I may write about the topic once more, to clarify more things.

Off we blog!

Distributed, Integrated or Networked?

2010-10-09T07:40:00.001-06:00

Distributed is a very common adjective. I usually tell my students to be careful when using it to refer to anything that gets in a network, which is usually the case. Even more, I tell them to beware of saying that REST or SOA are the answer to distributed needs, since it is not totally true. The question I receive is then: When can we tell something is distributed or not? Ok, this post will try to clarify the difference between the three terms you can read in the title, as they are very often confused.

Let’s begin by thinking about systems. A system is a set of related components that perform some interaction, following some rules. The components may be anything, from hardware to software, for virtual to real, from people to things. Now, a solution is the implementation of processes and flows that yield the resolution of (or helps solving) a problem. The solution may be a monolithic thing, or it may be a system.

If you take a system, and then you analyze what is it doing, you may find that it may not be good to have all processing done at the same place. So, you decide to split it, and place some parts somewhere else. Think of tasks, all performed by one big process in one big environment. You may need to split the system and send some tasks to be performed in other environment, by another process. That way, you can obtain some benefits like scaling, encapsulation, “replaceability” and better use of assets. Some say you have also reusability, but that is not actually true all the time, as it depends on how you split the system, and that is definitely not the goal of splitting. But, you also lose some other things, like performance due to increased interprocess communication (IPC). Fair. You have there a distributed system, a whole that was split but still needs to work as a whole.

Now, imagine you have two systems. Each system solves a problem. You may discover that joining forces, the systems can solve higher level problems. So, you decide they must communicate somehow, to try working together. Fair. You then create communication channels and you have an integrated macro system. That is, each system is treated as a whole that may be composed, reused, to create a bigger system. We need to keep the individual system’s inner workings a secret, and treat is as a whole, so we need it encapsulated, reusable and loose coupled. We use standard interfaces, and we use higher level processes. Also, the frequency of interaction is low. And of course, the user knows this bigger system is a large composed thing, where we know the individual parts. We call an Integrated system.

But wait: isn’t that the same idea as above? Well, not really. It may sound the same, but when implementing, and semantically, they are not. See, usually distribution requires the system to work as a whole, and thus communication is a pain the user should not be aware of. It must be fast, and should be tight coupled. Since the parts of a distributed system are parts of a whole, they know each other and that allows for faster communication if that information is used in coupling.

Let’s check an example. For distributed systems, suggesting Services through messaging as the communication channel may not work, since that actually increases the overload for communications. (Distribution with services seems to be a bad idea, an antipattern).

On the other hand, you have the integrated system. Here each subsystem is a standalone system. There may not be the need to have fast communication, but reliable and loose coupled one. There, a service fits, since we don’t need to know the internals of one system to integrate it with another one.

Fine. What about the Networked system? Well, that is also a system, but made as a network. A networked system has its components interacting as a network. That is, you have nodes and communication between all of them, you have routing to get to a node, and you have a very dynamic node topology. Yes, you can have integrated systems and distributed systems running in a network, but that does not make any of those systems a network system. You may have the same parts of a distributed/integrated system in one big machine, no network, and still be integrated or distributed.

In a networked system, you can easily have more than one copy of a node. Routing and balancing gives you the ability to increase and decrease the node number and thus scaling in place, incrementally, focused. You can add more nodes without affecting the ones in the network, and you can create subsystems with them. In fact, all you can do with a computer network you can do with a networked system. I guess the difference between integrated/distributed systems and networked ones is clear. No? Ok.

Let’s do it again.
- A distributed system is a whole system that was split to improve its parts, that needs fast and reliable communication, just like if the component being called is near at hand. Usually communication is direct, tight coupled. It has a high interaction between parts. Finally, the reuse of part is optional.
- An integrated system is the one made of whole systems largely distributed, encapsulated. Low frequency of interaction between parts, loose coupled (high cost) communication. The parts are made reusable, and the user is usually aware of the composition.
- A networked system, all nodes form a network. An application is a composition of node services. Topology may vary, dynamically, nodes may not be standalone components, and it needs low coupling and other messaging services like routing and balancing. Nodes can be added and functionality adjusted without much impact. Scalable locally, and may contain subsystems.

Ok. One big networked system is the web. Now, in the web you can have integration (B2B transactions), distribution (an enterprise with subsidiaries) and other networking systems (maybe mashups, or
Grids).

Clear now? Great! Based on that clarification, I pretend to write a future post: REST is tuned for networked systems! Stay tuned...

IWS 2010: The Why of REST expanded Abstract

2010-08-22T16:30:00.002-06:00

Why do we talk about The Why of REST?.

I recall the old days, when I first met REST. It was introduced to me as a revolutionary way to create web services without all the complexity of SOAP and WSDL. It was an airline system used as an example. Very simple to understand. And yet very suspicious.

I had lots of questions in the following hours. The explanation was somehow light, and it seemed to me like a normal web request, nothing similar to a service. Later I read the REST dissertation (I actually did some annotations that I was to put in a blog post, and never did), and found out REST was an architectural matter, far more than just a web request.

I kept on the searching and found several examples claiming to be REST incarnations, but they were clearly not. I found several implementations were REST didn’t even fit.

How so? May you ask. Isn’t REST a solution for everything? Isn’t REST an API, or an API creation technique? Isn’t REST a Services creation technology? Well, no.

REST was not created to be used on SOA as a service creation technique. The origin of REST is quite different, and it is tuned for specific problems found when creating systems like the web.

Actually, REST is an architectural style, and it is made out of a set of different sub-styles, each of them is there for a reason. Each style is a set of constrains, that has some benefits and some drawbacks, all carefully balanced to fulfill the need of a particular type of application.

Many tutorial show REST as something for the developer, when implementing a service on the web, full of tricks and tips on how to replicate the normal language and system operations over the HTTP protocol. Instead, REST should be seen at architectural level, where trade-offs are generated due to constrains, and for that we need to understand why each of those constrains are in REST to begin with.

This presentation will try to work on REST from the architectural point of view (Yes, no GET-POST-URI examples). Why is it a Networked System application so different, and why the REST constrains may help (after checking, of course, what those constrains are and how they impact the architecture). At the end, we will have some minutes to discuss in which SOA flavors does REST fit, and why is it not good for everyone. Several “whys” to explain, the big Why of REST.

What if you need Transactions and also REST?

2010-07-25T13:26:00.003-06:00

Some time ago, in a discussion at REST-Discuss, en example for transactions in REST was given. The solutions are usually complex and tedious. This is part of one of my posts, fixed a little bit to include the sample case.

Case: We have a travel booking service. The traveler usually selects a flight and then a hotel to stay. If the person cannot confirm the flight, the room booking should not be confirmed. Here, we have a transaction. It is all or nothing. Good.

How do you model that in REST?

Usual method is to have two resource, the flight booking and the room booking. And, there is also another resource you create, that is a transaction resource. That transaction resource contains the transaction data, either what to do or what was done. Finally, you commit or confirm the transaction resource, and all is committed.

Well. For this, I assume there are two different data sources, and we can commit to each one separately, but when a transaction involves individual transactions at each source, then you use the famous two phase commit. Each source is a participant, right? The airline and the hotel are two different companies, so your transaction is not simple.

We can have one server dedicated to the management of transactions. We can create the transaction object in it and add the steps and then ask him to perform (it can even perform at each step and verify all worked at the end, or rollback at any step if needed). The problem is, given we need to send to it, manually, all the transaction steps and actions, that idead may suffer some scalability problems.

On the other hand you have the client that needs to do all that processing to commit the transaction. That is, the client needs to be transaction-aware.

My feeling is that exposing the data entities as resources, and leaving to the client all the commit processing, is exposing too much the application detail. May not break REST, but adds unnecessary complexity.

In the example, we model, for the two phase commit, two sources, the airline resource and the hotel room resource. It is them implied that both are like databases, even more, separated database engines. And, the client will have to drive the transaction management to change data in both and then to commit. That is implicitly forcing the concepts of a resource, but still it sounds like REST.

So far, so good. Now, my question would be: should I need to do all that to actually reserve a package using REST? Well, thinking about how would I do it, I'd actually follow an online reservation workflow and see what happens:

a. I enter and search for a flight. System returns a list of flights and I select one. At this time a draft reservation is created with my flight in it. (Think a PUT of the empty reservation followed by a POST of the flight).

b. Then the system offers me to add a hotel reservation, and from the provided list I select one too. That is added to my draft reservation (another POST).

c. Finally, I add my credit card information and post a confirmation (Another POST).

This last action is served by server number 5 of 10 currently serving. That server 5 needs to complete the POST, and if unable, it will return an error to the client. Well, that server uses the draft reservation resource information to call a transaction manager to commit all changes. If it fails, server 5 returns the error.

That is totally opaque to the client, which only confirms and receives a yes or no to that request. Depending on that response, the client retries, updates the selection of flights or hotel and confirms again, or even desists and eliminates the reservation. Simple, ha.

But wait, that draft sounds a lot like the transaction resource mentioned above. Well, yes, but semantically it is not. The user adding reservations to a draft is not expecting each step to be on firm. The actual transaction occurs at the end, when we confirm all the bookings.

The difference in this process is that client is freed from knowing the transaction is happening. Resources are just that, no databases nor tables that need transactions and the client doesn't have to choose the use of single or two phase commits. You can scale since you can change the number of servers or transaction managers without touching the client. AND, each client interaction leaves the system in a stable state. Actually, this can be RESTFull too!.

So, if we can hide the complexity of the transaction, why do we need to expose that complexity to the client? I may do it if that brings some benefit. My question will then be, which benefits will I found from one implementation to the other one, or why one of them is not suitable for some particular business case.

Look that the actual transaction coordination is happening under the cover, at server level. That data hiding allows not only flexibility to adjust the process, but also reliability, since failed clients can recover.

Cheers.

Trying to Explain App State for REST

2010-07-24T18:09:00.000-06:00

I will try a different approach to explain the concept of application state, which is usually confusing and a recurring question from REST newbies.

I want to play blackjack. I have some friends and we all go to the online blackjack service.

Each one of us enters the application, which allows us to play BJ. The main screen requests our IDs. At the next minute, we all but one are ready to play (the one still not ready is because he forgot the ID number).

Let's see what happens there. We are, say, ten clients, each one is using the application, nine of which are ready to play while one is waiting for ID input. As you can see, there is one application, but each client has a different application state now. And each client knows its state clearly.

So, we left our friend finding his number and we started to request cards. We knew there was a dealer at the other end of the line, but we were not able to know if we were all playing with the same dealer!

Still, there was just one mass of cards. Yes, each card requested changed the mass. Still more, each request may be served by a different dealer!

Careful here. You may get confused. Requesting a card using GET is like asking the dealer to show you the face of the top card. You get a representation, but the card stays in the deck. In real game, you want the card sent to you. In REST, you will post something (a "draw a card" request message) to the dealer, with the actual deck URL. The dealer will modify the deck, say removing the top card, and will redirect you to that card. Here, the mass of cards is a resource, the dealers are the servers.

Now, can you clearly see the difference of the states? Each request will change the resource state, but that is not the app state. Each client has its own state of the game (each player knows its hand), and that is the app state.

The actual server state may be dealer working or dealer in a break. So server state is actually another very different thing.

So, what is the REST constrain here? Well, if you program the dealer to know the hand of each client, you are violating REST, since then the server is actually controlling the state of the app for each user.

The idea is the client controls its app state, not the server. So, even when the client wins the hand, it will have to show it to the dealer, saying I won. And the dealer will give me the lollipop gift. And after that the dealer will wait again for another request.

Each request will contain all the information needed to be totally completed in one interaction. The sum of all interactions is your final goal, the application.

Also note the resource can be changed in the process, but that is not an app change.

Why is all that setup good? Well, you can have thousands of clients that it will not impact servers, since they do not keep track of the hands. You can add or remove dealers with no impact on the clients nor in the application states. You keep hidden the resource implementation. And you got then a nicely distributed game, scalable and simple.

Hope this helps!

SOA discussion remembrances

2010-07-03T18:26:00.003-06:00

Back on the year of 2007, end of May, there was a post on The Server Side by Albert Mavashev that highlighted the growing complexity of SOA, why architects should take that into account, and stating that a balanced, incremental adoption of SOA was one good idea. Problem was his starting paragraph, where he says:

SOA (Service-Oriented Architecture) seems to be the buzzword everywhere!!! SOA will solve all the problems, everyone wants SOA -- or so I hear. Anyway, I remember similar claims when CORBA was coming to town. It was supposed to change the way IT does business. To me SOA is XML/SOAP based CORBA with evolving standards around it... I might be wrong.

That was a very long post, in terms of comments. The CORBA comparison acted as a flame bite, and we ended with 169 comments or so, of which 20 were mine!. At that time I didn't even have a blog, so this post contained the first discussions and comments representing the first attempts I did to expose my understanding about SOA. Reading them back today, I feel I can adjust some things, but must of the things remain the same, and I feel that when I wrote all that I was just a kid!.

I wanted to rescue my comments, what effect they had, and comment upon them in this post. Old memories!

The discussion was started already, with lots of enthusiastic people and some angry ones too. There was a comment line that included James Watson and Rodrigo Cana. James posted a link to a Thomas Erl article about SOA . The discussion was around if SOA, CORBA and EJB where the same thing under a different disguise. Rodrigo was saying:

I read the article, and I've never read that much bullshit about SOA.
Really the author does not have a clue about what SOA is about.

But I was relieved to read James Watson's post, that tried to clarify that SOA is not about tecnology to integrate applications. Instead, it is about how to manage the integration in a way to ease manageability and increase flexibility. So, if your "SOA" usage is increasing complexity, then you're applying the wrong concepts.

I read the article too. It seemed nice, but I felt that some misconceptions were given by granted, just because they were the current way of doing things. As a defender of concepts and theory against the mainstream, I answered to Rodrigo.

Rodrigo.
I'll jump in this line since I feel is the most sane in the whole discussion.

Yep, I read the article too, and although the guy has very good lines in it, it starts by falling into the same error a bunch on writers here
falls too. Just in reading the part of the vendor-oriented service-orientation, Thomas mentions something like "core expectation of SOA is its ability to harmonize and streamline diverse technical environments". Probably it may be a sell point for IT, but in fact thatis a side effect (not often achieved). The core expectation in an SOA is to model an application in the problem space using a business service metaphor.

In another line, Thomas says "Like traditional multitiered architectures, SOA is based on a model wherein solution logic is distributed". That is a trap. Services should be thought as complete business functionality offered no by methods or classes, but by services. Distribution leads to segmentation of code and business logic, and it is a technical strategy. SOA is about definition of business functionality as a whole (which may be internally implemented as a distributed thing). That is a bad approach, SOA is not to distribute, it is not DCOM over HTTP.

I Agree with Watson too. Although, he is still thinking in SOA as a technology issue pain reliever. SOA is about managing not technology but business. And the complexity is not about the concepts or technologies used, is about the impedance mismatch obtained by mixing them.

Agree, again, SOA is not a silver bullet. Jump into the buzz word bandwagon antipattern

Ok, I think the part of the expectation has changed a little bit for me.
Actually, I feel SOA is more in the integration line than it is in the distribution one, and I will blog about it some day. Back then, I got SOA more on business than as a technology for integration, I still think that.

James was back saying I might treating Erl in a way he didn't deserve, and actually I granted he was not suggesting but explaining how things were done. So, sorry Thomas.

Later, I kept reading another branch, where there was much discussion about the Hype and about CORBA as a SOA sibling. I answered a comment by the author post, Albert, saying:

Hi Albert.

Hard to keep on reading all the messages (jumped late to the discussion). But I think I have a couple of interesting points I want to highlight.

1. Goosh. The debate between WS people and Corba people is always the same. I do teach Web Services course at university, and I see the same ideas back and forth in my students. WS IS NOT distributed components.
CORBA is about a broker pattern for distributed components, and WS is about business service metaphors and how to communicate with them using messages. Apples and Oranges.

2. SOA refers not to a technology, as many pointed out, but to a supra design in the problem space. Yep, nothing related to the implementation, but to the business level. Services are aimed to architecturally designing the business processes using the Service metaphor as the building blocks. The problem here is designers are not architects nor business people, but plain developers that, by the way, worship OO and RPC.

3. Getting to your point, I understand CORBA suffered from complexity in implementation, and lack of governance tools. But Orchestration (different from governance and even choreography) probably was not needed (there was the broker in the center). Services are other things.
The messaging system is the one that requires orchestration, choreography and governance. Services need to be designed, as business entities, with lax interfaces that are business oriented. No RPC. And so, you need also governance of business processes. Since the actual market sells tools to OO developers, they are oriented to the RPC governance and emphasizing the call flow and not the messaging manipulation.

By this I mean that the tools should have two levels, the one for developers, that should be in the cubicles implementing the utility classes for the service implementation, and master designers with business background designing the services per se. And then, IT support controlling the messaging system pitfalls, and business people controlling the business flows.

If you want to compare CORBA and SOA complexity in governance and orchestration, then I would say they are different. And I would say lots of people don't really know what are they calling SOA anyway, which makes it even harder.

On another branch, Jose Romero and James Watson were discussing loose coupling (the original branch idea was that CORBA was not the same as SOA, and later that SOA was a hype worse than Web 2.0). Well, Jose was mentioning a relation between coupling and synchrony:

In response to james, there are 2 categories of coupling : (data) type coupling and temporal coupling. Type coupling is the most obvious : the type of exchanged data induce coupling between client and service. Using literal xml document preserve the structural semantic and allow for flexibility/variability without adverse effects (in fact infoset would be enough to convey data). Temporal coupling is more subtle. To support message exchange pattern other than traditional request/reply service must support asynchronicity. It doesn't imply that the message exchange pattern must always be asynchronous. To reduce temporal coupling between client and service allow for a more robust system : partial and transient failures don't block the whole system

I jumped in:

Jose.
I would thing there are more that only 2 types of coupling. Actually, coupling is a generic thing that feeds from several variables.
The "synchronicity" you mention as temporal is one that is not as trivial to spot. Data typing is another. RPC "exchange" pattern (I hate to call that), is another one, since it forces a semantic in the message body, even a pre-stated structure of operation and parameters with a return value. The transport may be another one, and there is where HTTP nature, for instance, is confused with "synchonicity".

How come? HTTP is not a messaging transport. It is a Hyper Text transport, based on the REST idea of distributed resources and local states (although REST came later). No messaging anywhere. Thus, since HTTP operations require a response, and that means they need to be a blocking pattern, doesn't mean that the messaging that piggybacks on it should also be. As Hohpe mentioned in a conference, there is no message that goes and comes back. There are actually two messages. The fact that HTTP is as good to send a message in a request and bring the other message in the response, shouldn't tie the blocking or synchronous pattern to messaging as well. IN other words, you can have the request response without needing a synchronous call.

Lastly, and that is something I always say to my students when talking about products, I see no client-server relation in SOA messaging. I see an origin of the message and a destiny. That final target contains a service that consumes the message. A service is not method exposed. It
is a cloud of coherent functionality that is activated by messages, and that may send a message back to you.

I love that vision

Then, Constance Eustace opens a new branch ranting about all the guys that just say SOA is not this and that, but no one states what it actually is. Excerpts from this:

"SOA is distributed procedural programming. Services are stateless request-response calls on arbitrary destinations. "

"SOA seems to be going after a high-level design of simple procedural calls and data retrieval/updates/deletes/inserts, with a "unified" calling interface, if one can call the current state of service/webservice acronyms unified or simple."

"SOA is an attempt at simplicity that has failed:
- technologically
too many different standards
no agreement as to what the hell it is
overly complicated (failing its primary mission: simplicity)"

To this, I obviously answered. In my response I stated a couple of things that may not hold totally correct. For instance, I say services are stateless, and they may be but that is not actually a requirement. I also said, SOA was an architectural Style, still there is no definition for it. I will try to defined later in another post. Here is my answer:

I agree with several things you say. I disagree with the overall idea.

Yes, as the hitting the moles, each time someone tries to define SOA (a mole getting out of the hole) someone crashes a pole in the head of the poor thing just because it didn't understand SOA.

There is a difference between what SOA is and what SOA has become due to development incompetence.

SOA was confused with distributed procedural (or object oriented too!) programming.

Services are stateless, but they were forced to be RPC calls.

SOA was forced to sell the idea of a simpler, unified, platform independent interface, which is not (a service IS NOT an interface).

SOA was sold out as simplicity when it was just a way to architecture using the service metaphor.

SOA has a different vision, not huge, not grande, not even new.

CORBA is based on an architectural pattern (also called Style) that is the very well know Broker pattern, that was sold out as an implementation (the same all vendors try to do with SOA).

SOA is an architecture style, where you model using services that are named functionalities accessible by messages. Is not panacea, and is not simple. That is why architects (and I mean Enterprise Architects) are the ones that should deal with it, and not Java programmers.

Finally, simplicity has nothing to do here. Trying to do SOA without thinking in architecture and with the actual tools, creates stovepipes systems, impedance mismatch, little monsters that make overly complex to call a method using a service metaphor with asynchronous messages.
Creates chaos.

The question would not be what is SOA and what is not. The question should be Why are all these people trying to make the square SOA vision fit into a round hole? It is hurting a very good abstraction that, by the way, is not good for everyone.

Finally, we would never agree on what it is, since every kind of "hammer" looks at it as its very own kind of nail.

Following that rant, Jordan Xu justified using an art analogy that I found amusing. He said:

"They want to take IT into the realm of abastract art. Ever feel like being in an art museum, standing in front of huge canvas with a big red square in the middle?

Anyway, they can't justify their "Chief Architect" title without an "architecture" to preach for. Since they can't talk about anything concrete and get caught with their empty rhetorics, they hype up something abstract like SOA. No one can say they don't know what they are talking about because nobody knows what it is"

Interesting. What Jordan describes the architect's do is an antipattern called Intelectual Violence. He may be right for some architects I've heard of. Still, I answered to the point that SOA was an abstract thing, and actually to the abstraction concept:

Well, Jordan, at least we architects sometimes do agree about something.

This is not about who is right and who isn't. Is not about art (abstraction is the elimination of details to model a reality, not Picasso's paintings).

By the way, abstraction eliminates details but keeps structure, which makes what someone says a very concrete solution.

I have a hard time trying to make developers crack their shells and see what other people are doing. I'm being unsuccessful to bring a Hibernate user to think abut XML databases, or even Object databases. There are lots of java developers I check their code and see procedural programming, no OO design anywhere. The same is happening with services (forget SOA). Even other people teaching at the university see Services as objects exposing their methods, and there is no way to change that mind. If they cannot understand services unless I talk to them in terms of objects, then they will never understand SOA.

At least, my new hope are my students. Some of them are starting to investigate a cross model representation. Actually it is hard, they say, and the tools are not of use, but at least they have the concept pure.

By the way, I'm a hard core developer, differenec is I swallowed the right pill.

There is actually a paradigm shift that is needed for modeling and implementing. DOSE for instance (Document Oriented Service Engine) is still in napkins drafts, but the idea is not to use stubs for web services, but message brokers instead.

Then Jordan came with an interesting take on interfaces that started another discussion flow. He said:

"The concept of breaking software systems into independent, reusable service components has a major flaw. It does not address the changing needs of the business environment. Many of these services, such as "process payment", may be considered permenant, but the devil is always in the details. What are the input and output of these services change all the time. By fixing the interface to the services, we often made them very inflexible. If even the reusability of plain objects are often difficult, how can we be so sure that we can define services that are constant(in all details) and thus reusable?"

Reusability word was there. I know everybody things on services as reusable things. I have had some discussion about that in other forums, and the idea actually is tracked back to when objects ruled the earth, and thus my peers said services were an evolution from objects! Still, I don't like the reusability thing. I feel you should create services to be composable, not reusable. I posted this:

I see a repeating idea about SOA, services in particular, that doesn't fit in what I have learned from my childhood as an architect. I will explain myself.

I recall the old Object oriented days when all the fuzz was about reusability. The idea was clear: separate the code into coherent and functional complete things. Then a problem was found: the interface. You could not reuse the thing because of the interface not being suitable, or requiring things not in the new problem domain.

Then a service came. It was the idea of having a business functionality (not an interface, not an object) able to accept messages and offer a business value. It has no reusability concept, the fuzz was actually about replacing fixed interfaces with a simple port. And somebody said:
Let's implement it with SOAP. And SOAP was Simple Object Access Protocol, an RPC oriented wrapping protocol used to access distributed objects. Wow. Wait a minute. Something is not working here. I know objects were supposed to accept messages. Nobody uses that. But a service is not an object. A service has no methods or parameters. Definitively, it has no response. There is no interface per se, just a port where you deliver your message.

Now, the message is not SOAP. SOAP is a wrapper. The real thing is in the body.

Once the service has read from the port the message content, it will now know what to do. And it may decide to send another message. As simple as that. But as difficult to grasp for an object programmer, that it was reduced to a YAMTCARO (Yet another method to call a remote object).

NOw, can somebody tell me why a service needs a method name to be called? SOAP had stubs in the client, to simulate the remote object as local. Services are not object, stubs have no place here. Makes no sense. You need a port object to stream your message. Simple.

Finally, since a service is just a port, and the service semantics (which are not technically describable) is hidden behind the service cloud. You can create a versioning system underneath the cloud, so old messages can still be accepted and new messages, that represent new business needs, be processed as required. Service cloud still as white as before. It may have a new port, but old ports may be still there.

Enterprise architects? Their job is to have one eye in the problem space, and the other in the solution space. They will work with business people to design services scope (business driven), and work with designers to technically create the structure of the cloud.

Hope this helps to understand why I say all that I say.

The concept of services with a uniform interface is also base at REST architectural style. There is the idea of that being impossible? REST is a working thing, and that interface is possible, sure it is. Well, Jordan told me that concept was far away from mainstream understanding of SOA. Made me feel as the only one with that idea:

"William, your vision or interpretation of services in SOA is very different from the mainstream. The most popular description of services is a set of independent, reusable functional units that can be "reassembled" to form different applications. I am not sure how many architects will agree with you."

Oh, well, time will tell.

Meanwhile, Karl Banke took the idea of the port and managed to create an analogy to RPC. Very interesting use of a metaphor (I saw it later again, mentioned by Guido):

"Well, if a service is just a port, there must be some semantics for the port to know what to do with the message is receives. While there might be multi purpose messages, they are even harder to maintain than single purpose messages. So, almost all messages to a port will essentially be "single purpose", like "make a booking", "transfer money", "get account statement" etc. Lets call this the "function name" or "message type". Of course the client will usually have some kind of return message, which, for the purpose of this discussion, shall be called the "return value". The inner content of the message shall be called the "argument". What is so different from a method call after all. Essentially nothing. For the sake of simplicity, we might even have some fun, and create an EJB or Corba Object, that we call the Service Port.

What you are doing is "de-abstracting" away from perfectly valid abstractions, arguing about the service cloud that might have a silver lining."

Interesting vision. My students did ask the same question. At the end, sending anything to a port can be mapped to an RPC call model. Humm, I'm convinced that is not the case. Actually, document passing has not more semantics for the port than to receive a document and deliver it to the appropriate reader. And sending back a response is totally up to the reader, by the way. So, I tried to answer that with an example:

Service is not just a port. A Service is very complex thing that has a simple mission: to provide a business functionality as a service.

How do you communicate with it? Using a port. You send a message to a port. That the message body contains an XML structure that is thighly coupled to reassemble a method call, is not of the Service business, it just doesn't care. That is, using that inflexible analogy makes your service miss those important features like being weak coupled, but it is your call.

Now, what separates an architect from a designer/developer is that the first one works with the service image I told you, business. The second ones will take that vision, see how can they publish that port, what do they need in the message or in the port itself to determine functionality, create a message processor that will route the requests to business logic and other levels, and then finish the work. The easiest way is the lazy programmer way. They put all the business logic in one class, create a method to publish it, create an XML schema that reassembles the method call, and use the port to get that XML. In other words, they are downsizing the metaphor to an RPC that may be far less expensive if done with EJBs or even CORBA.

In other words, it is valid if you want to make your service look like an exposed object. I see no point in it, though.

Now on your examples: "make a booking", "transfer money", "get account statement". I see them clear as fine grained method calls. The first thing you hear in the street is to get away from those. They are expensive. Your need to create course grained services. The next example may seem a little extreme.

Service: Loan service.
Functionality: Actually, the service is able to process loan requests, payments, notifies due loans and you can ask for a loan status.
How is it done?: There are four document types, each one has a particular schema: LoanRequest (input), LoanInfoRequest(input/output), LoanNotification (output) and LoanPaymentNote(input). Any input XML document may be sent to a port located in www.willyloans.com/loanport or to the email loanport at willyloans.com. There, an XML processor will determine the schema type, and send the actual data to the loan analysis department, to the loan payments subsystem, or to the loan service desk system. You can guess what each system does, and what each document may contain.
You can create four WSDL if you want. All pointing to the same address. Just make them document style.
More functionality? Just add it. Another schema and that's it.
Versioning? No problem, several XML databases already support that.

What do you think?

That answer drove another set of comments about scalability using strings and James Watson saying it makes no sense to talk SOA in technology terms because SOA is about business. I couldn't agree more with that. Still, I tried to answer all those topics:

1. A WS with a string parameter may fit for a very tight concept of service. You send a document, and a document does not contain just a single data, but a structured or semi-structure sets of information. You can compress that into a string field, no discussion, but that I wouldn't do.

2. Then, a document is not just a couple of parameters. eXML standards and industry are creating XML documents that are in the order of several megs. Pack that into a string. Parse from XML to String, then encode it to add it to XML SOAP, then decode it, and un-parse it. Make 100 concurrent users try to send the same document to the service. You will receive less that 1 tps. That is a scalability problem. And since RPC is a blocking mechanism by nature, the problem is reflected in the clients too! I have several stories about clients timed out.

3. Still not sure Services are stateless. I know communication between them is. A service is not only a process, it is functionality and may have a state, although they are implemented using a stateless object (EJB).

4. Guido is totally right, Services scale by concept, implementations draw away the scalability.

Still, John Brand was interested in my product comparison. He said that going from 1 to 100 users sending one document decreasing the TPS was not scalability problem but a performance one. Also, he did not see the impact on the architectural perspective. Here are my ideas on that:

1. Probably the example was extreme and confusing. Performance problem is when your "expected normal" load is returning a low TPS. Scalability is when you want to increase the load, using more users or bigger documents for instance, and the expected growth lowers dramatically the TPS. So you are right, the example was missing the increase factor. I assumed 1 user and then grow to 100.

2. It depends on what is "architecturally significant". From an architect point of view, I may want to use a service oriented interoperation between two of my partner companies. If I know both companies work with EDI formatted documents, and I see that my architecture design will work only if those are kept in place, I may decide to dig to that level and indicate the use of EDI payload instead of XML. To me, that is "significant" to deep my hand on. As such, I may also indicate to designers that we must use message queues instead of XML-RPC, for instance, since we are dealing with As/400 machines and legacy code.

3. About the scalability, that is a quality property that architects usually look at. I know that the service I'm asking to create is a very sensitive one. And although I have just a few clients now that will use it, I foresee in a month I will have 5 times the clients, so I need it to scale. That is "architecturally significant" too.

4. Finally, I may want to handle the spec, or simply tell to the designer: I need it to scale to 100 users and 10 meg messages. I wrote an article about the need of messages with a full, huge payload. The idea is actually to separate the bulk data from the message, what you don't have to do is create messages with bulk data. message are not good carriers of huge payloads. Use raw xml over html or keep the payload in some place, but not to "call" the service.

Further down, I found Paul Beckford. Paul is a very nice guy to discuss with, and I like very much to discuss and drinking good coffee. He asked a very small question:

"Are you talking REST?"

Ha! My Service loan example reassembled much of a REST implementation.
Interesting. Still, back then I was convinced the web services built using REST were not web services, where just RPC in the URL. Now that vision has changed, because I have seen the correct way of doing services using REST constrains. Still, that comment was a summary of my REST position:

Well, Paul, I'll answer that one.

REST is another word several people use, but may not be clear of what it refers too. Chapter 5 of Fielding's dissertation explains REST. It is an architectural style used to create distributed hypermedia systems.
Doesn't sound like services, right? Because it was not created for them.

REST is based on a state. Imagine resources in the net. The resource as a state that may be changed by simply performing a reduced set of operations (post, get, put and delete). The resource is accessible using its ID (URI). And you work with a representation of the resource.

For most people, REST-like web services is simply calling "services" using arbitrary XML documents and http. It's not that easy. There are lots of confusing ideas there. For instance, there are people saying services are stateless. But resources is REST are not. The communication should be stateless. Confusing, right?

Web Services Architecture spec has a view of the architecture where the services are seen as resources. So, we may even say that Web Services using SOAP are actually REST-Like!.

REST has not idea of messages. And it is for hypermedia. That means that web services using message queues or some other transports have no meaning in REST-world.

Services as resources. Services do not have a representation, they have a port that is actually the web resource used.

When you request a page in the Web, the representation of that page (the HTML code) is self-sufficient. The browser will paint it without trouble. But with an XML document, you as a developer must know what those tags mean to actually do something, it is not self-describing as REST request.

For this, and for several other dicrepancies, I thing that REST style web services shouldn't have that name, since they are a missreading of the REST idea. Confusing, right?

Well, Paul, I think the short answer is that the above ideas of services, (extracted from the spec, not the vendors) are actually implementable using document style web services, (SOAP and WSDL), and also using what has been call REST-like services. We can even implement them using message queues, SMTP, or plain TCP/IP sockets.

Funny, huh?.

This may go to my blog someday

Paul answered some very nice things. I understand I usually go too academic, since I'm of the strict definition, correctness driven type of guy. Still I understand practice is not always as theory dictates. Just like the quote I found in Nicolai M. Josuttits' SOA in Practice Book, that he reproduced from Laurence Peter "Yogi" Berra:

"In Theory, theory and practice are the same. In practice, they are not.".

Paul ended his comment with this:

"I agree that many do find REST confusing, but I don't believe the underlying idea is confusing in itself. It is one of those ideas where less is more IMO. I find that people (including myself) struggle to come to terms with the less.

The central concept behind REST is the idea of a uniform connector, which seems to fit your "document style" "distributed architecture" idea pretty well (I am avoiding the use of the term "web services" since I believe it as been polluted in much the same way as SOA). As for SOAP, there is very little uniformity there. Hence I agree you could implement REST with SOAP, but to do so would miss the point.

Anyway, thanks for a rare ray of intellectualism on TSS. I have followed this discussion and felt no need to participate up until now. I will continue the discussion on your blog if you don't mind.

It is nice to meet someone with something thoughtful to say".

And to my blog he went, with very richful discussions!.

But the perils didn't stop there. Guido Anzuoni stated, with much of reasoned logic, that all went down to XML again. After that, many other comments started talking about implementation. Even lokesh Pant cherished the change from 10000 feet view to implementation one. All was interesting. Then I found Karl Banke dissecting my response and commenting about what I was saying. He thought I was "seriously on the wrong track", at least about my way of looking at services as business functionality units. Actually several people told me that, but I feel relieved to learn there were other not thinking like that, like Nicolai.

Well, I answered Karl, again:

"Ok, Karl.
I think you an I agree on several things. I have a problem sometimes when people doesn't clearly understand what I'm referring to. Let's see:

1. A business functionality cannot be abstracted in that way. There is a wrong concept between developers, that assume abstraction is talking about business needs, opposite to implementation. Abstraction is creating a model of something, with less detail. You can abstract your implementation using a model of classes in UML, or you can abstract a business process using a business model. Problem comes when developers try to mix the two.

2. A service offers some functionality. That should be clear. The developers implements all he needs for the service to work. Simple.

3. "Object perform(Object)", as I say, it's your call. That assumes objects in the platforms, assumes a response, and assumes parameters. That is coupled to me, you only "loose" the particular object type.

4. My notion of an architect is much more complex. I teach at the university that course. An architect looses some detail as a master designer, to obtain it back as a business analyst. As I said, Architect is not the oldest programmer in the company, it has competence in six major roles or subdomains. But I cannot comprise six moths of lecture in a paragraph.

5. Far away from what I have said in this thread. Services are not for everyone, they are not panacea nor silver bullets, they need more an architect than a java developer, and they won't be useful or even different from what we already have until the mainstream stops buying the vendors interpretation of a YAMTCARO, and requirement analyst stop asking the developers to create services just because everyone is doing that.

6. As you see it, it may be right. "Make a Booking" seems to me a perfect task for a Booking Service. You can also add a another task to cancel a reservation and even request an availability report. All of them perfect tasks for a Booking Service. Those are called "operations".

7. Lastly, I mention a "complex thing" not because services are difficult to create. They are just not as simple as creating a method with a string parameter.

Hope this helps."

And. OOOPS, it started a discussion between Paul, Karl and Guido. Paul was telling he thinks all my ideas were actually REST, and that I was confused, and posted some examples of why with REST the universal connector (as he called it) made interfaces that didn't change, as opposed to WS-*. Karl was saying we are were discussing bizarre things since technology has nothing to do with never changing interfaces. And Guido stated the uniform connector added just another layer of abstraction, but the coupling of was still there since callers and providers need to agree on a contract anyway.

Well, I guess all that sounds interestingly correct. But there are some missing points in there that made them correct and missing the point at the same time (hummm). For instance, in REST you discover the flow from links in the hypermedia documents. The same with WSDL, problem is people uses WSDL as a static thing, design time. Karl was correct, but REST is not a technology per se, ha, and some technologies do force changing interfaces. Of course we need to agree on something! Guido is correct, but the coupling is not in in the interface, but at the business level, where we want it to be.

Ok, the comments went down to discussing what a uniform connector was and why CORBA IDL was not for the job, and what benefit did that bring to the implementation (to have a uniform connector, that is). It is clear in the REST dissertation, though.

After a lot of comments discussing who was right and actually saying the same thing over and over again, I said good bye to Paul stating that Uniform Connector was all that I mentioned about the port and just sending messages to it.

Yep, it is a good thing to remember old days.

Owning the Architecture

2010-06-17T07:19:00.002-06:00

One of the main principles in Software architecture is the fact that there is always an architecture. That is, there is no system made without an architecture in place. Good or bad, the architecture is, conceptually, the system structure, the organization of components, their relations and the relations they have with the system’s outer world. So, you cannot have a system without it.

The problem, discussions and disagreements come when we address the architecture actual handling. There we may find the position of letting the architecture “emerge” from the code written. Architect’s are sometimes decidedly marked for extinction, since they are not needed for development, simply because they do not “code“ but perform a Big Design Up Front that is useless at the end of the day (that is what that position holds as true, I disagree of course). Other positions often refer to architects as facilitators for high level design and decision takers regarding global and important issues. Yet, to others, architects are just highly experienced developers that should be helping coding, another level just above the senior developer.

The position I now describe here is the one I heartily believe is the one all architects should adopt. (Just because it is the natural one for handling the architecture.)

First, let’s see what’s been done so far…

How do you deal with the system’s architecture?Well, you can:

Develop, code and later on see what was created.
Define it (design) and then try to create it by development.
Forget about it. Focus on code.

As you can see, architecture will be always there, but it will grow wild in all three cases. Yes, case 2 also has problems since almost 100% of the time the resulting architecture is different (sometimes substantially different) than the one defined at the beginning. Common sense dictates that we should at least foresee the architecture from the beginning and start building it with code, keeping an eye on what that code is actually producing.

Now, no matter what you do, you end up defining an architecture. Either by planning, by designing, or by directly coding it. At the end, the architecture is defined by someone. The question is then who? (architects? Developers? The client?) And then How? (coding? Designing? Deciding?). Common sense dictates again that all those roles are involved in the definition and creation. That means there should be a coordination, and a high end control of what is going on (not dictating, but controlling). That role is, of course, the one of the architect.

How does an architect control the architecture definition?Well, we have several stereotypes applied to architects, that we can recall:

The documenter. Architects are seen as document generators, that are happy writing designs and other documents nobody reads.
The technical Project Manager. Here the architect is the one managing the project at the technical levels. A technical lead. Sometimes a technical director. Some others a technical puppeteer that holds the developers strings.
The Ancient, Know-It-All. This seems to believe that years will create a good architect, with unsurpassed knowledge. Many time this architect is seen in a Ivory tower, and some others it just holds the answer to all questions.

None of the above stereotypes really deal with the architecture control. Documentation may be an important tool to keep updated information of what the architecture looks like in a precise moment in time. There should be enough coordination and direction capabilities in the architect to manage the building process. And of course, there should be enough experience to make the correct decisions, coach, mentor and provide advice to coders and designers. Still, documents are not the final goal, nor the architect dictates directly what is to be done, nor he knows all and should not be at all difficult to approach.

So, we can say the Architect Owns the architecture. Owning means basically being responsible of:

Knowing what the architecture is, looks like, in a moment in time.
Adjusting the principles, decisions and rationale based on the feedback got from the development process (this includes the information that actual coding returns, plus stakeholder feedback).
Rectifying actual coding and design if that breaks the rationale of the defined architecture.
Make everybody included in the development process (that also includes stakeholders), participating on the adjustments and the architecture visualization.

It is clear now that the architect’s role goes far beyond that just imagine a proposed architecture at the first phase of the development process. It is responsible for the real architecture, the one built, at each step of the cycle. For that he will need:

Visualization of the real architecture. It is out of the scope of this post to discuss how to do that, but one process I should mention is the “In Deep, Vertical Peer Review” that involves all three levels of review (strategy, tactical and operational) in all the phases, instead of just one review per phase.
Rationale and goal validation. This process is very important, where we can validate the goals are still being achieved even if there are changes in any of the levels. For that, rationale is extremely important.
A trade-off process. Of course, it is impossible to always achieve everything. There should be a trade-off process that evaluates the options and resolves for the one that keeps the goals as unchanged as possible, and results in the least cost. This of course involves all stakeholders.
A cross refactoring process/redesign. All adjustments will change things. Refactoring is a way to improve things without changing actual behavior. The redesign may change behavior. The idea of the process is to try keeping the changes at the lowest level. That is, try first to adjust only code, without changing design nor strategic/architectural decisions. Then go up. The strongest change will require the architecture redefinition, with possibly the highest cost.

All this applies to all phases, including the quality assuring, deployment and acceptance.
How to do all that in different contexts and projects require a whole book. So far let’s simply summarize that:

Architect deals with many tasks, including but not restricted to documenting, managing and deciding.
The architect’s responsibility is to get an architecture done (meaning, built) satisfying the stakeholders goals. This requires continuous verification of goal and business value achievement.
Performing all that goes far beyond just coding.

Conclusion: Owning the architecture means being responsible of its construction, taking care of all the complex processing that development requires.

An Evolving Architecture?

2010-06-11T21:58:00.002-06:00

Written by: William Martinez Pomares
Published on March 1st, 2009 @ 04:42:42 pm , using 619 words

Neal Ford from ThoughtWorks, recently wrote an article that is the first of a series related to architecturing and design called Evolutionary Architecture and Emergent Design. The first article is a revisit to the architecture and design concepts. In it, Neal describes the evolutionary architecture with these lines:

However, just because you can't allow architecture to emerge doesn't mean that it can't evolve. If you have created a flexible architecture and taken care not to create an irreversible decision, you can allow it to evolve over time as new concerns appear.

Now, I wrote a comment on TSS where I think the "Evolutionary" name may be applicable to the architecture concept I have:

In several discussions I had had, the idea of an architecture is repeatedly presented as a bunch of paper with fixed ideas that will not come true after all. I argue that should not be true.
Granted, a bunch of paper may define the "architecture to be", but the architecture is not ideas but the actual structure of working software at the end. So, Architects must define the final goal not as an stone written truth, but as an ideal outcome which will be supported by principles and guidelines to succeed. Those guidelines should show the path of adjustments to the ideas depending on the construction evolution. And all that is aimed to solve the stakeholders concerns.
In that line, and architecture is what you get at the end, and you set your goals and guides to drive the solution at the beginning, and make your decisions not as late as you could but as soon as they are in the best moment to be taken. (Yes, it is not the same). Under that idea, the architecture is something alive, that will change over time while it is being developed. It "evolves", and that is what I like about the article.

Still, something feels not quite right with this. And that is the idea of thinking we have an architecture from the beginning to evolve. Yes, what we have is a goal with a nice idea of what the architecture will be, but no real architecture (remember the real architecture is what you have once the project is done).

Now, let's clarify this further:

Evolution: the process of removing and adding features or improvements so the new product is better. The natural evolution may be by chance, but the induced evolution is pre-defined.

Evolution Rules:Evolution always follow a predefined set of rules. For architecture, these are the principles and guidelines defined at the beginning of the project.

Designed Architecture: The description of the Architecture-to-be that solves the stakeholder concerns.

Real Architecture: The one you have in every system, even if it is not documented. Of course, you need to finish the system first.

Using the above terms, you can have a set of concerns, the you design an architecture and a set of principles and guidelines to construct it. Those principles should allow for modifications based on a retrospective analysis of the results so far, at each step of construction. Once construction is over, you have the real architecture.

The thing that evolves, is the actual architecture design, the real architecture is emerging. Unless you construct whatever you think of first and then start changing all that is not working to improve. There you can have not one, but several architectures, one is the evolution result from the previous one.

Finally, you can have, as Neal mentions, an architecture designed to evolve (with flexibility included), which is different of what we are talking here. That would be an evolving system that has an architecture that allows it. Tricky!

REST and the Wagging The Dog Syndrome.

2010-06-10T23:12:00.000-06:00

Written by: William Martinez Pomares
Published on April 9th, 2009 @ 11:06:45 pm , using 559 words

Commenting on Tim Bray’s blog post about the use of Post for REST, I came to a discussion about actually how much RESTFull is an API, and more specifically if it should be RESTFull at all.
You see, many people is trying to get their APIs into the REST world, and some people like Dilip Krishnan believe the reason is to gain "simplicity, discoverability and serendipitous reuse of the solution". Still, as I read through all the comments on how to "imagine" the app so it fits REST, I feel a greater sense of something not right about it.
When creating a solution, you may want to analyze first the functional requirements, and then the quality properties. The ones noted by Dilip are great, but I’m not sure all apps that hit the web every day need to accomplish them. Furthermore, REST is not the only way of getting those attributes! Let’s explain a little what this famous style is so you can understand what I’m saying.
The REpresentational State Transfer style was born as an architectural style to represent/improve the modern web implementation, as Roy Fielding explains in his dissertation. So, REST is an architectural style, and that means it has a set of constrains that the app needs to adopt to obtain the benefits. But that architectural style is not built from scratch. It is built by composing several other well known styles like Client-Cache-Stateless-Server, Layered-Client-Server or Code-On-Demand. Any of those can be used separately and all their pros and cons will be cast upon the solution.
Now, this is a special case where an implementation is taken as a base to develop an architectural style that improves it. Usually, you have a solution to build with a set of requirements, and you look for an style that better fit into the problem. But once you start changing the problem just to adjust to an architectural style, you are actually “Wagging the Dog” there. And it seems to be a contagious syndrome.
So, what does it need to have an APP to be worthy enough to use the style? Well, first

- the semantics should be the ones of the style.
- Next, the app should try to solve a similar issue the style is solving, and third,
- the app’s quality attributes should be the ones the style is promoting.

If some of the three is not true for your app, and you still want to use the style, I may guess you are trying to get the buzzword title instead of the real benefits.
One little clarification at this point: The web is one implementation that follows the style, it is NOT REST itself. What do I mean by that? Creating an app that runs on the web doesn’t make it RESTFull. Using HTTP for your API won’t make it either (I wanted to clarify it since all comments are about HTTP and its semantics and it seems everybody confuses the implementation with the style). You need to read the dissertation to learn all the constrains, but even more important, to learn if the style is for you or not. You may learn HTTP is not a required protocol, but one constructed that meets the constrains. You can build your own! (and make it standard).
William Martinez.

Glossary

2010-06-01T17:42:00.000-06:00

Written by: William Martinez
Published on April 26th, 2008 @ 06:31:45 pm , using 448 words at here

During all these discussions, in this blog and other forums, I have found some semantic dissonance. This means some people use the same word and even the same concept, but actually understand different things from it.
So, words like architecture and design mean paper to some and process to others; they may be normal, unnecessary or even offensive. This is a little post that tries do define some of the words I use, just to be sure anybody that reads me understands what my meaning is.

Design: As a verb, is the process of decision making towards the creation of a solution. As a noun, it is the one word name of Design Description.

Design Description: The description of the decisions taken toward a solution and their rationale. It can be a document, image or even multimedia.

Code: Sentences, written in a particular language, that describe the solution. It can be at any level, and may (or not) be executable.

Strategic Design: Design that focus on architecturally significant issues, whose majority are global decisions described in an intensional way.

Tactical Design: Design that focus on tactical issues, whose majority are local decisions described in an intensional way.

Micro Design: Design that focus on operational issues, whose majority are local decisions described in an extensional way.

Domain: Is the set of concepts and relations that define a reality. It is not a description, since those concepts may be infinite or unbounded.

Abstraction: Is the result of abstracting: process of elimination of detail without loosing the essence of the concept being abstracted.

Model: Abstraction of a reality from an specific domain.

Modeling: Process of describing a possible reality in an abstraction level.

Architecture: System Structures, the externally visible properties and the relations between them. NOT PAPER.

System: Independent structure of components interrelated.

Software Intensive Systems. Systems where the majority of the components are software based ones.

Solution: Part or whole of a system that solves a problem from a domain.

Meta something: Meta means beyond, above of, self-referring. Using those concepts of the suffix, we may say meta-modeling is modeling models. Meta-data is data about the data. Meta-language is a language that allows us to describe languages. Meta-comments are comments about the comment. And so on.

Now some notes: from the above, we can say interesting things. For instance, modeling is not a paper generation process, but a description process. Modeling can be use to describe a solution, thus it can also be interpreted as coding. And code, it is not just writing executable lines (in fact, they are not executable, they need more work).

When time and other discussions require it, I will be posting more glossary terms.

Why Web Services Failed at SOA (and why REST may fail too!)

2010-03-09T11:13:00.000-06:00

First, let us review what is the concept behind services, with particular mention of the web services themselves.

A service is a business functionality exposed by a simple interface. Note that it is not an object, nor a method, nor a module, not a procedure. It is pure business functionality.

Services are consumed, not called, nor executed nor invoked. We define the service consumption as the interaction between agents, a client and a provider, which will communicate using a messaging system, where the data unit is a document. Consumption may be local or remote (meaning a service may not require remote communication).

A service may live in the web, where they are modeled as resources, identified by a URI. If not living in web, as suitable endpoint implementation should be provided.

A service is described at business level with a contract, where a glossary of the terms is agreed upon, and the semantics of all business concepts and processes are clearly stated and known by interacting parts (this is important). The service implementation is defined by the service name, its contract, an endpoint (unique entry point/port) where to send messages, the operations the service offers, the message interaction flow, the message composition (including the document definition) and optionally a binding information.

Nice definition. What is wrong with it?. Not much, actually. Just that when Web Services Architecture group decided to make those concepts into a standard, they did a great job, but also committed a couple of fatal mistakes.

The first one was the actual selection of SOAP as the messaging protocol. Yes, SOAP is a big mistake, not because it is SOAP, but because of what it was created for. You can read here a very complete explanation of the SOAP origins by one of the people that worked on it, Don Box. There, we can understand the main problem to solve was creating a protocol to better support RPC (actually RMI) in a object distributed system.

Yes, SOAP (Simple Object Access Protocol) was not created to support services documents and messaging. It was created to provide a technology neutral data/class definition system, plus supporting remote method access (that actually meant Remote Method Invocation).

So, why is this a mistake? Well, unless your idea of services was, from the beginning, that of distributed objects and the exposition of their methods, SOAP was no fit for a Services architecture. Why is SOAP there, not sure, but my common sense tells me it was not its place. If the idea was to support document oriented messaging, an RPC protocol was the weirdest decision I can think of. You will need to tweak the protocol to support what it was not supporting before. The only reusable aspect is that it was meant for HTTP transportation, which by the way is not a constrain of the WSA (it is the only implementation though).

History tells that soon after, SOAP was modified to allow document style definition, relaxing the literal RPC format. The SOAP acronym meaning was actually changed! So it is not an object access oriented protocol anymore, but a service oriented one. That should prevent people using it for RPC, was the hope.

Ok, if that was fixed (SOAP adjusted), we may have no more problems with that, right?
What is the second mistake?

Well, a subtle one: they kept the RPC idea in SOAP. Backwards compatibility? Not sure, but the RPC literal was there, in SOAP. Worse, the WSDL included that option and WSA did not discouraged the practice! I may think it was because there were tools supporting this and the idea was not banning anyone from using them. So, people started (or continued) using RPC and calling them services!
Is that wrong? Well, a little:
a. The problem is RPC is not Services,
b. RPC does not share the service metaphor,
c, RPC has different implementation side effects,
d. RPC usually forces fine grain while Services try the contrary
e. RPC couples (per method name, parameters and interaction)
f. RPC is in the IT domain, while services should be in the business one (this is a big one).
g. RPC usually works in the distributed environment, but services not are always distributed.
h. RPC forces request-response flow, while services may have any combination of flows.
i. RPC is usually blocking, synchronous. Services are asynchronous per messaging basic concepts.
j. RPC parameters usually require serialization of objects (to be passed as parameters). Services do not even know what objects are nor the parameter concept, they deal with plain messages.
k. In HTTP, as a particular implementation, RPCs mapping of concepts adds overhead to the call, while Services uses HTTP just as a transport and should not add more overhead.

Ok but, what about that? In case the RPC was kept due to tools, the existing tools at that time may be kept as RPC tools, and new tools to work with services and their concepts should had been created. Easy solution.

But that didn’t happen.

Actually, the tools took over and started to fight back! There was almost no support for document style. There was an RPC style non encoded. There was no standard way to represent a document (that you can feed the service call with). Yes, the tools claim they supported document style. But when I did some research projects to compare the tools, I found the services created using document styles didn’t work at all! I created a service, then I created a client using the same tool, and the client failed talking to the service! I made it work by manually structuring the XML document to have the first element named as the method that implemented the service! It was RPC all along, and since nobody complained, I assume no one was expecting to work document style. Well, there were a few.

But the worst comes afterward, with the creation of the Wrapped and Bare modifiers. They are no were in the WSDL nor the WSA. The wrapped is a way to use document style, but doing what I did manually in the lines above: formatting the XML to reassemble an RPC format message! You know what, testing the BARE modifier yield what I suspected: didn’t work.

That means the tools and server vendors are still selling RPC, creating RPC “services” and RPC clients. The ones that supported the document style, did a great job making people stay away from it, like Axis, whose example of RPC is a couple of lines long, but the same example using document style is gross and long, not even I could read.

Ok, granted, the web services implementation are just RPC in disguise. So what? We’ve been using RPC all our lives and we are happy with it, right?

Well, SOA is a style that looks forward to have its structure and components made out of business functionality. Under that assumption, we can build lots of additional features, like business composition (business processes), orchestration, governance, etc. But, for that we need services as business functionality, decoupled by an uniform interface. RPC adds coupling to the mix, since we are adding the object or library concepts (for many, those two are the same!) , the method invocation interaction, the data coupling in the arguments lists, and the stiffness of non-variable method definitions (just change one argument and all breaks!).

Point (f) above is important. All other features of SOA are built on top of the concept of a business functionality exposed as a service. Governance, composability, visualization, all those work with services, not with function calls. So, as mentioned in point f, what surfaces is IT, not business, and thus the features may not work as expected.

Add to the mix that, to actually create the services, you must be a programmer. Since they reflect the RMI concept, we assume distributions, method and parameter passing knowledge, etc. Even more, the actual code is converted into exposed services, bottom up approach. This will create services shaped as the actual legacy implementation, which may not match the actual business at hand. The top down approach, which should start with the definition of the service at the business domain level, is discouraged. Tools offer not help there (or very little), and only a few architects follow that path. So, the RPC influence is breaking the possibilities of actually taking advantage of the Services metaphor.

Can I say that the whole failure of SOA as a agent of change, an encourager of business over IT in systems, a provider of business valuable concepts like business processes, governance and product composing, is just the intrusion of RPC in the whole dance? May be, but there are other causes as well. Still, SOA may not raise from he tomb if no paradigm shift is achieved.

Wait a minute. REST is in the title, it is another problem in your list? Or is it just the enemy of SOA?

Well, not really. REST is another architectural Style. It was made for a very different problem and I can assure you it is not the replacement of SOA.

It is in the title because of two reasons:
1. People think the problem of SOA were the Web Services using SOAP,
2. and also people believe REST is a simple way of creating the next generation of services.

So, they are adding another point of failure to the new SOA by trying to construct SOA using what they call “RESTFull Services“.

Why is it a fail point? I’ve heard REST services rock!

Well, first I own REST an apology. You see, actually REST has nothing to do with this. The problem is with the wrongly named REST services, which are no more than an API based on HTTP. Now, there could be REST services, but they are much more complex that the ones most people use. And bigger. So, let’s simply call those “Not-So-REST Services” as Web API Services, (WAS? Nice acronym, a provider for one of those would be a WASP!).

Well, our WAS are, as usual, more oriented to functional calls (yes, RPC). There could be a way the WAS achieves a truly service meaning, when using HTTP as the messaging transport, and the Hypermedia as the documents. Yep, it is a nice match. But that requires people thinking on documents and interactions through messages, and they are not doing that. If you check, it is the same old RPC story again.

So, what to do then? Are we doomed, I mean, is SOA doomed?

Not really. We need to understand what a service is about, and provide a suitable implementation that allows developers to work with it without mapping. I mean, why does a service need to implemented as an object method call? It breaks the metaphor and thus breaks design and implementation, making it flawed.
I’m working on a DOSE (Document Oriented Service Engine) implementation in Java, WAS and WSA oriented (WSDL 2.0, that is), with no RPC. The programming model should be quite different that the proxy and RPC model, plus a set of articles to show how design using document orientation for services should be done.

So, stay tuned.

Longer Abstract for TSSJS presentations

2010-03-09T11:03:00.000-06:00

DSL and Testing: using JSR in the Polyglot Architecture.
Long ago, we started working with a Client, PushToTest, that has a tool for testing. The main idea of this tool was the ability to repurpose unit tests into functional, load and Monitor tests. It has also a proxy to record all transactions against a web page, so the transactions could be replayed later. At some stage, the intermediate language chosen to script tests and record the transactions was Jython.

Over the years, it was learned that not all people knew Jython, nor Python nor Java. So, other scripts were added, adding support for popular solutions like Selenium and SOAPUI. Still, we found there were cases where developers had code snippets written in their language of heart, by that ruby, scheme, and others. Then the goal of the tool became being as flexible as we could make it, So, JSR 223 was added to the mix using the external implementation in Java 5, and later using the integrated one of Java 6.
The next step in evolution was to create a communication environment, in a way such that complex transactions testing was possible using at each step the appropriate technology. For instance, we may use … to process data files and execute updates to web services using SOAPUI. Later, we will verify the data is updated in the database by issuing a web report request using selenium. To do this, all scripts should share info, communicate. We created DPLs (data production libraries) able to provide sample data to the running tests, across threads (we simulated multiple user using threads). Some scripts were able to execute methods from objects created in other languages, and we were able to create complex scenarios by launching different groups of threads with multiple roles (for instance, some consumers and other providers, or some posting blogs, some others searching, others registering and other running reports).
To create such a complex system, architecture was the key. One part, what we call the polyglot architecture, is a simple abstraction of runners that allow a central controller to run different tools and languages.
Furthermore. For some of our clients, we were in the need to create simplified languages or scripts. We could do some of those things on java or jython, but we would be forcing the client to learn java to actually modify the scripts. In those cases, we created simplistic languages with a handful of operations, that may be run with a particular runner. Yes, those were little DSLs, and the magic of JSR 223 allow us to integrate them seamlessly into the polyglot architecture.

Want to learn more about this? I will try to explain what a practical DSL is, how to use them for testing, and how to implement a JSR 223 engine for them. From that to the polyglot architecture, it is just a copy step.

What is a REST API, anyway.
Have you ever wondered what an API is? Well, API may be a common used word, but many just give the definition for granted and had had no second thought about it. Now, REST is an architecture style for networked systems. Given that, how would a RESTFul API look like? This is the question we will discuss in the chat. Actually we will see that many of the APIs have not much in common with REST, and that you may be creating a simple web api rather than a full REST one.

Coding vrs Development.

2010-02-02T19:29:00.002-06:00

Agile taste for working code over documentation may be misleading to some. As seen in many posts and discussions, it seems that it means no documentation at all for some people. For another group, it means you just need to start coding, nothing else matters. This post is to differentiate plain coding from development, and to explain that working code is more that just executable sentences.

Let’s start by saying that code is the description of a solution in some sort of language. I defend the position that the description of a solution may not be executable. That description is to be read sometimes by the machine, some others by humans, and some others by both. The description has to have an abstraction level. That is, based on the approach we use to describe and to whom, it may be machine language (assembler for instance) or a high level code like a DSL. And we don’t just write ones and zeros because we will need later to revisit that code, and be able to read it and to make changes. Bear with that in mind for later.

Development is not just coding. It involves finding the why in the business needs and decisions, deciding the how we will solve that, actually describe the solution (to stakeholders and to the executors), and verify that solution meets the needs.

One point to clarify here is the executors concept. A solution is not a chunk of code. A solution may be a complex system, that is software intensive. That means it is not just code, it usually contains processes, people, and many other elements. All together are the solution. And that system is to be built in development, not just the code part.

There are several other tasks that should be performed during the development process. As we deal with humans, you need to have people management. A group has always issues, problems, questions. Also, humans need material to work with. Meetings, information, computers, and many other assets. During the development, several decisions should be made, and not just about code: deliverables, QA scheduling, assignments, feature modifications, priorities, etc.

People needs to communicate. Creation of communications channels, management of those, coordination, moderation, etc. Some voices say all that work should be done by developers themselves. Not so sure about that. Some other say that is not needed, we just need code writers to write code. Not sure that works either.

So, so far, I believe development is far more a complex process that just writing code. And coding is far more complex that just writing an executable set of instructions for a machine (Description of a solution, remember?) Thus, I proclaim that coding is not the same as developing. Let’s continue.

Going back to the coding definition. Some people think that a non executable code is waste. Specifically, design is seen as useless documentation, wasted time. Here then I start to defend what we may call design.

Non executable code is that one that describes de solution, but may not be read be a machine and may not be executed by it. As we mentioned before, the executer may not be a machine, but let’s assume the executable code is executed by a machine not to mess with concepts. Some of that code is the one generated as strategic and tactical description, aka, architectural description and design. If it is not executable, why do we need that kind of code? There are several reasons:

The ones to design are the developers, why? Because it is a description of the solution, and thus code.
Executable code is often done quicker when the ideas and major decisions are already made in the design.
When code is done, it is usually reviewed and almost always changed to improve it. Describing at design level is quicker than writing low level code, and thus fixes and improvements can be made earlier, saving time and effort (changing executable code, large portions of it, may have a great cost and impact).
Once done, the executable code should be verified. Checking against a high level description helps to verify. If it is different, then it means either the decision at high level or at low level need a review. It is important that both descriptions are kept in synch.
Why doing the work twice? Because they are different works. Description at high level is used at some time, and the low level description later. But both are used. Missing one means there is something in development that may have more cost later. See next point.
Development is never finished with complete code delivery. There are always enhancements, additions, changes. To decide why, how and what, it is always important to know what is done, how and why. Reading machine code means almost impossible changes. Reading executable code is possible, but at high cost. Reading a high level description, that was kept in synch, is great.
Design is not a documentation process, it is a description process that should be used through the development. If design outcome is not used, the it is waste. Design that is done trying to be complete before operational coding, it will also be a waste of time. Design should be made in regular chunks, not to little ones that misses the tactical view, not so big so that you end up making decisions with not enough information.

So, given the ideas above, I want to proclaim that used design is actually working code, and thus, if well done, complies with Agile thinking.

REST Fans Categories

2009-12-12T13:30:00.002-06:00

First of all, let’s clarify a couple of things.

1. This post is not intended to list “bad ways” of thinking about REST, but to list the actual way people really think about it. Some of them are perfectly correct per REST constrains.

2. From articles, presentations, discussions and the like, I extracted commons ways of thinking, and tried to categorize them. So, this is a totally subjective classification.

So, let’s start.
People know about REST. Many are trying to get into the REST world for the first time, some other have more flight hours, still not so many had read Roy’s dissertation. Even more, some of the ones that actually read the dissertation, may think about REST in different ways.

Usually, when in a discussion, the same concept may be understood differently by the participants, which yields in non-useful conclusions. This may be actually true about REST. That is why I composed the following list of “ways people understand REST”. The list may help the reader to find out where he stands in understanding REST, and how may other people understand it. Also, it may show how some people is using REST.

First, an cautionary word: the names I use are not intended to make fun of people, but to be easy to remember.

The list is divided into three major groups: The API makers, the Mappers and the Fad Followers. This last one may seem a bad one, but it is actually a normal one, trust me many people fall in it.
Then, each group contains categories. Those categories may be present in more than one group, since the characteristics may fit both groups. Finally, people may fit in more than one category, since the work is complex in the world of REST to reduced to just one characteristic.

Here is the list.

Major Groups 1: API Makers.
People in this group I find them everywhere. Most of the time they have a system, usually not built thinking on REST at first (REST as an afterthought), and they want an API created to access that system. They usually think about REST like an API making technique or recipe, for the web.

Categories:
- URI Jugglers. These are the ones that think REST is all about creating URIs, and nothing more. So their discussions are solely focused on URIs, and their presentations are about URIs definitions. One way of spotting this category is when the application API is documented as a URI construction process using templates

- RPCers. This group think REST is a way to map RPC in disguise using URIs in a web API. The most of them don't know they speak RPC at all. Easily spotted when you see the API defined as a set of calls using a URI construction. Most typical is a URI format that includes the actual method name or action, followed by several parameters. There are many of them coming from SOAP based web services (the RPC kind), that made the change believing is easier to define the RPC call in a simple URI than using the complex SOAP envelope.

- Exposers: This type is repeated below. Those are the guys that think you need to expose things in REST using resources. So REST is an API for exposing things on the web. They usually look for objects or domain entities, but mostly processes or methods, and expose them using a URI formatted call. The slight difference with the RPCers, is that the exposers may not use RPC, but still expose the processes and entities through an API.

- CRUDers: Another repeated group. They think REST is a web API for CRUD. The similarities of the HTTP operations and the CRUD ones, plus the reference of resources as Data elements, support this way of viewing REST.

Second Group: Mappers.
This other group may use the API idea, but they actually think REST is a way of representing things and thus the work to be done is to map everything using that new way. Interesting?

Categories:
- CRUDers. Again, the idea is that CRUD operations can be mapped naturally to HTTP operations, and that makes it RESTful.

- HTTPers. They believe REST is HTTP. Deep enough. This category is hard to spot, since many people talking about REST implementation may talk much about HTTP, but think REST is far more than just the protocol.

- Exposers. Again too. They usually try to map all classes, data entities, elements into resources, and then call their systems RESTful. This category includes people that take into account REST constrains, and also people that just maps the domain to resources.

Third Group: FAD followers.
This is a group that is identified by themselves. That is, they expose what they think explicitly. Usually, they tend to follow a lead or the current business idea.

Categories.
- Standard Haters: Here you have all those that think Standards are evil and that REST is an anarchy where you have the freedom to do whatever you like, so they follow REST doing whatever they want.

- KISS lovers. These are the ones that like things to be simple. And someone told them REST is easy, so they follow doing easy things with URIs. There are lots of URI jugglers in this group. People here may not follow all REST constrains when they get complex enough. The idea is to keep the API or system with not much complexity.

- Servicers. They think Services is good, and someone told them REST is a way to do services without SOAP. So they follow. There are several RESTFull service creation techniques. Many RPCers in this category too.

- BuzzWorders. This is a vast majority. They like buzz words, so they follow REST just because it is cool and all people talk about it.

Is there some one I'm missing? Well, yes, probably the group that knows REST as it actually is and understands it. That may be a one person group (yes Roy).

What do you think? Do you find yourself in any of those groups? Which other category may you add?

The Agile missing point and the Waterfall Illusion.

2009-11-25T20:39:00.004-06:00

What if everybody is wrong? Well, not everybody, but majority.

Reading Kelly Water’s Post about “Agile Project Management: Avoiding The Waterfall” I came to think about this. Almost nobody claims today that they are following waterfall, because of the very bad connotation that word now has, but not actually because they understand what are they doing. Same thing for Agile followers.

From the post, I rescue that any process that has phases reassembles a waterfall. From actual presentations, blogs and discussions, Agile often means just following methodologies like Scrum, Lean or XP. Now I question: are we right about this? Let me see if we are, or not.

Waterfall is a term, it seems nobody knows who coined it, related to a phased methodology. Dr. Winston W. Royce wrote about it (without given that name), describing an ideal form that was not so good . The phases were the requirements gathering, analysis on those requirements, system design, coding, testing and operation. The ideal was each step may iteratively cycle with the next down and the next up, in little steps. The flaw was it was usually the case that testing will not just go back to coding, but it may go up till design, and from there, jump back to requirements gathering again. That flaw (the large back jump) was caused not because the phases, but because the testing phase is where the actual product is finally validated against design and requirements.

The problem is, there is no solution to this. We can dilute or minimize the impact of the testing findings, using several techniques, but we cannot avoid it. At any level, team or individual, the same steps are repeated: I see what it is needed, I analyze what is possible, I think on how to do it, I do it and finally I check if it works. That can describe the process of developer creating a simple three line method! That is, phased development is there, everywhere, and that is not the problem. The problem is when that phased process is done in a way that the impact of facing reality with needs is too big that multiplies the work and the cost of the back jump.

So, the waterfall illusion is when you think that rain is not a waterfall, simply because the raindrops are so tiny you don’t see them fall individually. But they fall anyway. It means, people that think eliminating the analysis and design steps and jump into coding breaks the phased process, I’m sorry, it doesn’t, it may avoid a large body of water fall into one place, but will make millions of little bodies fall everywhere. See the point?

Ok, there is another idea into this, and that is to make the water flow up. You start coding, with no idea of what you need, and then the phases are completed backwards. (Hey! If you want to start really backwards, then start by testing! Oh, yes, we have TDD). The code will make a design “emerge”. The next logical steps if to create requirements that match the design. Well, I guess that does not work. In that case, the requirements are still presented before coding. So the new phases are: requirements, coding, design, testing. Or, requirements, testing, coding, design, testing, coding, design, etc. Anyway, we always have phases.

Now, Agile. In the Agile Manifesto the idea was to value common sense upon theory. That is why “individuals and interactions were valued over processes and tools, working software over comprehensive documentation, Customer collaboration over contract negotiation and Responding to change over following a plan.”. Nowhere on those lines there is a mention of chaos process over phased process. Where is the missing point? Well, actually in three places.

1. Comprehensive documentation was assumed to be the design’s product. So, design must go, correct? Well, that is not true. The idea behind this is to prefer creating working software rather that documenting. Actually, creating working software requires you to think first on what do I need to create software for, how can I accomplish that, think how will I do it, do it and then check if it works. You can do all that without opening a word processor, right? Working software is not any software, but the software that delivers the business value, and that is not produced just by hitting keys.

2. Customer collaboration over contract negotiation seems to define the negotiation to be a non collaborative task. Actually, any customer involvement implies a contract (or mini contracts), which is the definition of what to do, and who will do what. Collaboration does not mean the client will bring coffee, or will help writing the code. It means active communication to shape the bush with little cuts. The client says the overall figure, then supports indicating the little branches you may cut further. Contract and mini-contracts.

3. Responding to change over following a plan does not mean going into chaos. The following a plan refers to follow a line without looking at changes in time. That is flawed thinking, the plan may perfectly be to respond to change in an organized way, validating each time to improve the next one. The plan is simple: we decide as we go, following these principles.

All the above is in the principles.
“Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.”
“Continuous attention to technical excellence and good design enhances agility.”
“Business people and developers must work together daily throughout the project”
And
“At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.”

Now, let’s talk about project managers and architects.
I understand many agilistas think they need to disappear. Yes, YAGNI. Architects are document creators and thus have no business in Agile. Project Managers are plan followers thus they are not welcomed in Agile world.

Of course, that is not correct. I mean, the stereotype many people has about architects and project managers. In a team, you need different roles. If you have only coders, and all people just code, you have an incomplete team. Roles are not fixed either, you can have four, ten, twenty different roles in a team. Each person performs a type of work, needed to complete the main goal.

So, does that mean Architect and PM are roles? No, they are professions, and each one plays a role in the team. And no, an architect is not the one to write design diagrams, and the PM is not the one with MS Project under his arm telling everyone how to type its code.

An architect is the one that owns the architecture, that is, knows what the structures and organization of the functional elements and other architecture issues, are. Not only that, he is responsible of assuring the business goal is met, either changing the code that builds the architecture or changing the architecture to match the code.

The PM is the one in charge of measuring everything, keeping an eye on evolution, times, resources needs, organization of meetings, pending issues, team problems, process glitches, and a long etc, so coders wont need to worry about those things.

Closing this long post, I can summarize:
1. No, there is not only waterfall and the good guys. Water is falling in all projects, we need focus in minimizing the bad effects.
2. No, agile does not mean following a methodology and getting rid of good guys like Architects and PMs, but understanding the common sense of the manifesto and apply it to improve development.

Cheers!

Quick Post Edit: Reading at this, you may think I believe all Agile people thinks the same. Well, no. I was practicing agile principles even before the name was coined, and find out those are misunderstood by a lot of people. Fortunately, not all of them! So, Agile is the way to go.

Willy's SOA Manifesto

2009-10-25T13:07:00.008-06:00

Yep, my own. Although it may have some shared ideas, this is not a group agreement, nor the savior of what is now known as SOA (which is what is being done so far), nor something made for others to follow. It is just what I believe, what I follow, and what I teach each time I can.

So, here it is.

SOA MANIFESTO

1. Commons.
a. SOA is an acronym composed of the words Service Oriented Architecture, which relates to an architectural style that is based on the rules of the service metaphor.
b. A Metaphor is an application of the figure of speech concept to the IT design discipline. Using metaphors in architectural design actually refers to observing the rules of behavior of a particular entity in the business domain, and mimic them with an architectural element.
c. A Service is conceived as a cohesive (and coherent) business functionality, technologically neutral, offered through a uniform interface. A Service is a plain, homonymous metaphor of a business service.
d. For an architecture, Orientation describes the guidelines, principles and decisions that are based on the rules that govern the metaphor behavior.
e. The Architecture refers to an actual style that defines the architectural constrains, suitability and consequences of using it.

From (a) to (e) :

SOA is an architectural style whose components, constrains and principles
are driven by the service metaphor.

2. Delta
By definition, and in contra-position from some usual implementation, SOA IS NOT:
a. An Indirection Layer for interoperability.
b. A Decoration Wrap for legacy code accessibility.
c. A Component from a whole that represents one architecture instance
d. A Service Group or container of services
e. A Business Process Container.
f. A Modernization Agent
g. An Object Distribution Technology
h. An Antagonist of REST
i. A ROI salvation
j. A Way of Life, A Philosophy of Doing or a Trend (It is an architecture style!)

Corollary of (j): SOA cannot be killed, it cannot just die.
It can be suitable for a problem or not.

By definition, and in contra-position from some usual implementation, Services ARE NOT:
a. Necessarily Web Services as defined in the WSA.
b. Ways to implement RPC nor RMI.
c. Exposed Methods or Objects.
d. Defined by the process they execute
e. Necessarily Stateless (YES, they should not necessarily be Stateless).

3. Principles.
Whenever there is the opportunity when working with SOA, people should
a. Favor Business Value exposed in the business domain OVER Business Value exposed with IT domain.
b. Favor Composability OVER Integration through Interoperability, AND favor Integration through Interoperability INSTEAD of Distribution.
c. Favor Metaphor rules INSTEAD of Tool rules
d. Favor Pure Document Style INSTEAD of RPC/RMI style. (NO WRAP TRICKS ALLOWED)
e. Favor Messaging OVER Other communication options.
f. Favor Business needs OVER specific IT practices (like optimization or flexibility)
g. Favor Entity documents OVER commanding documents
h. Favor Versionable documents OVER Structurally static documents.

So it is written.

William Martinez Pomares

Notes on SOA Manifesto

2009-10-16T14:43:00.004-06:00

In a recent discussion at InfoQ about the news of the upcoming SOA Manifesto, I had discussion with great people, including Jean-Jacques Dubray and Steve Ross-Talbot both renowned guys in the SOA world. We all talk the same, maybe at different levels or on different realms. Still, I promised Steve to put my two cents about SOA. Well, here I have a couple of ideas about it, too high level, but trying to clarify at least the three words that conform the acronym.

Well, here they go.

Many people talk about SOA and Services, but I’ve found in presentations, articles, case studies, working with tools vendors and looking at real implementations, that not everyone understands the concepts quite right. So, I have a couple of posts (here, and here) and a short video in the Architecture Journal (SOA Myths), explaining the types of perceptions people have about the concepts, and why I think they are wrong. I assure you it is a good list, interesting to read. So, that would be my first contribution.
Now, let’s check on the Service concept. As I mention in the posts, to me a service is a cohesive (and coherent) business functionality offered through a uniform interface. And it is technologically neutral. What does that mean? Well, a service is a plain, homonymous metaphor of a business service, just like the loan service in a bank, or a delivery service of you mail office. As a metaphor, it is driven by the rules of common services, with contracts, processes, protocols, controls and evolutions. Its actual implementation may be whatever, but it should not surface the definition, meaning users of a service do not have to know how the service was implemented to use it, and should not have to learn anything else apart from the rules and protocols of regular business services interactions.
Now, Orientation. For an architecture, orientation describes the guidelines, principles and decisions that are based on the rules that govern the metaphor behavior. That is, the service one.
Architecture refers to an actual style that defines the architectural constrains, suitability and consequences of using it.
So, SOA is an architectural styles whose components, constrains and principles are driven by the service metaphor.
Service implementation is out of scope. Service metaphor implies business domain. The implementation domain should not surface into the business view, not affect the rules of the metaphor. This is something almost all of implementation fails to accomplish: to use a service you usually must know implementation details, and those are the ones that actually avoid flexibility and change.
The service contract requires a semantic agreement (standard shared business semantics are a must), functional specification, pre and post conditions and the description of side effects. All that is technologically neutral.
Lastly, and just for the records: to me, reuse is not a business concept, but an IT one. So, Services should not have as a goal the reusable property. In business, the property is composability. The key is to be able to create higher-level services from composing lower level ones. Reuse is a side effect, and should not be the base for ROI, because it will never make it!

Hope this is clear.

William Martinez Pomares

Styles, Pattern and Idioms

2009-10-12T12:18:00.004-06:00

From a recent discussion with Duncan Cragg, related to his proposed FOREST, a get-only-rest-integration-pattern I came to think that it was good to create a little post to explain the difference between an Architecture Style and a Design Pattern. So, let’s do it.

It was Christopher Alexander, in his book A Pattern Language: Towns, Buildings, Construction. (Oxford University Press, 1977) that coined the Pattern term to denote ideas or solutions that were proven to be successful and widely used.

Well, that book was about buildings, and Alexander was an architect, of the building class. When mapped to the IT field, we have three levels at what we develop: Strategic, Tactical and Operational, or Architectural, Design, Implementation. So, we have those “ideas or solutions” in all those three levels. For Architectural level, we have Architectural Styles. For Design level, we have Design Patterns (or just plain patterns) and for Implementation level we have Idioms. You can read the definitions in POSA1 (Pattern-Oriented Software Architecture:A System of Patterns from Buschmann et al, Wiley 1996).

The style name is discussed by Roy T. Fielding in his dissertation about REST. He actually dislikes a little the name since he says it represents more a particular way of doing things (that singer’s style) rather than a general way of doing things. Coming from the art world myself, I can say it has that other meaning: a style is a way someone follows, and thus “followable”. Given someone is so original that creates a new style that others follow, does not take away that followable quality, only makes it richer.

So, as you know, the architecture of a system is the organization of its elements and their relationships, guided by principles. When defining a style, you define what types of elements would you use, what do they do, the relationships, and principles to guide their construction. A Simple example is the Client Server style. It identifies two architectural elements, the client and the server, indicates what each one does and who will the interact. Not this is a global solution: the whole system works Following this style.

BTW, Roy defines the Styles more like a set of constrains, coordinated, that define the elements and relationships. Actually, REST is defined that way in this dissertation.

When defining a Pattern, you work closer to implementation. There, you define a solution in the domain context, identifying actors and processes, relationships and results, aimed to solve a localized problem. The example here is the Factory. To solve the problem of needing different but similar objects depending on some parameter, the factory proposes the creation of the right type of object. Note this is in an object oriented context, and the solution is given in terms of objects, and it is to solve a particular problem in the whole system, not to define the whole system. Finally, note this is an idea that can be implemented in any OO language, and particularly in Java you can do it using inheritance or interfaces, meaning the pattern does not impose an specific implementation.

The Idioms are little patterns related to languages. These are proven ideas of how to accomplish things in one particular language. Say, to use StringBuffer in Java to concatenate strings.

As you can see, in general, patterns are good things, but quite different depending on the development level you are using. Now, to define a pattern, you need to provide at least five things, per Nick Rozanski and Eoin Woods, in Software Systems Architecture: Working with Stakeholders Using Viewpoints and Perspectives (Addison Wesley 2005).:

A name
A context where the problem and solution may be presented.
A problem to solve
The actual solution, given at the appropriate level
The consequences of applying that solution.

So, it is important to check all that before defining a new pattern, or before applying it. This last part is very important, since you may end up using a pattern that is not suitable for the context, for the problem or that has unwanted consequences.

Hope this helps clarify the concepts!

Talking about DSL

2009-09-23T19:58:00.002-06:00

Originally Published on March 11th, 2008 @ 07:30:36 pm

Hey!
Back after several months!
It is incredible the amount of work a change in your job position brings in its tail!

But for now, let's talk about the title.

I was recently reading a great blog from Michael Stal (which I'm adding to the interesting blog's section). The latest post is referring to Domain Specific Languages (DSL).

It is interesting, the list of examples of what can be considered a DSL. Being as attached to formality as I’m, my take is on the D and the L of the acronym.

First, it should be Domain driven. That means this “tool” will work inside a domain scope, referring to the elements that make sense inside that domain. If used as a communicator (as I will explain later) the reader may have knowledge in that domain. For instance, if you explain your solution to a manager in terms of resources, time and cost.

The second is the language constraint. That is, DSL is a language: a grammar and a vocabulary (plus maybe rules to expand that vocabulary). The language is used to describe things in sentences. Those things could be the problem or even the solution. That description can be read, understood and even executed.

I recall a very old issue of Dr. Dobbs that talked about mini-languages (or was it micro-languages?) that were script languages in essence. An example was the MS Excel macro language of that time. Specifically, the macro language had the typical constructions for conditions and loops, but the vocabulary was focused on cells, rows and columns, selections and texts, formulas and styles. The Domain was very explicit. It also mentioned another languages, some derived from C. those where subsets of the C language with some tweaks. Were those DSLs too?

Probably no one will say C or Java is not a language, so we have the L from the DSL combo. Now, which is the Domain for a language like Java or C? Or even for those little subsets of C? I guess we can find one, very broad, but it will not fit into the idea of specifying a particular domain and focusing on it. Thus, my bet is no, they are not DSL.

Now, let’s go a little further: people use DSL all the time! I’ve known programmers everywhere that use encodings in strings. Have you done that? A string that contains a list of values separated by a “separator” (colon?) and the first value is the code of the operations, the other values means something and such? Have you hear of the interpreter pattern? Those are DLS in a String! Micro DSLs, something executable and some other time just to display.

Wait, here is another point: Does this DSL need to be executed? It may not! Actually, a language is to describe things. ADL (to describe architectures) will no be executable (probably). So they may be just to describe, to be “Human-Interpreted” languages (it may also be machine readable to validate it, but that may not be a requirement). What about a VDL (View Description Language ) that will allow the architects to describe the viewpoint of a solution in a language in the stakeholder domain ? It may be graphical, but not required. It can be a simple subset of natural language. Let’s invent the DSD (Domain Specific Dictionary) as a glossary for the DSL vocabulary.

The example that just comes to my mind is Bool’s language. It has some symbols, but when you read them is sounds like natural language, but focused on logic! A AND B OR C IMPLIES D.

Will.

Reference: Talking about DSL

WSDL 2.0 - a REST Friendly Language

2009-09-23T19:51:00.002-06:00

Originally posted 07/20/07 @ 12:47:55 pm

W3C has published the Web Services Description Language (WSDL) Version 2.0 Recommendation. That means we can talk about a new standard now.

From history, you can recall SOAP was not a standard by committee but a de facto one, and WSDL 1.1 was a member submission done by Microsoft and IBM. Seven years later, WSDL 2.0 comes to see the light.

I mentioned before in this blog that WSDL offered four types of messaging, a combination of style/encoding parameters. There was an RPC style, and a Document Style, and for a backward compatibility, RPC could be Encoded or Literal.

Also, WSDL allowed a extension to define the transport, but it was just to say we were going to use HTTP, and that to send a SOAP structure as its payload. It was fixed to use the POST method, and it was also fixed the headers contents and such. Not much flexibility.

Well, WSDL 2.0 changes that. It now offers only one style: document. With that you can simulate your RPC calls if you want. Also, it allows to define the content of the document using an XML Schema, but that doesn't mean you will be sending XML through the wire (more on this later)!
But it doesn't stop there. WSDL 2.0 expands significantly the HTTP binding spec. Now you can specify if using GET instead of POST, indicate if you want you parameters (defined in the schema in types, remember?) to be encoded in the URL! Even, you can indicate that you want multi-part messages!

How friendly is that to REST? Keep on reading...

I know. Most of the REST community may hate anything that has to do with Web Services using its standards. But I feel that enterprise systems should not go in the wild with the REST complexity. Which complexity? you may say, if REST is the easiest thing. Well, yes and no. It is simple, but not easy. Everything has a complex component, and in REST it is the process of manually reading the contract and creating the interaction in code.

Yes, it is ok for simple parameter passing (two or three) even for some simple XML documents. But if you need to work with very complex XML and interactions, there may be a problem. Enterprise services may not be two or three parameters, and it may not be fine grained services. A way to describe all that so developers free themselves of understanding and coding XML to start thinking in business logic and processes, is a plus.

Will WSDL 2.0 that allows complex and simple things to be described, be a choice for that need of description? Who knows. What I know is that WSDL 2.0 seems to allow more Restful services descriptions, where they apply.

For more information about WSDL 2.0, please read the primer
For an article that explains Rest in WSDL 2.0 with more detail, please read this one from Eran

William Martinez Pomares.

Reference: WSDL 2.0 - a REST Friendly Language

An ORM Story

2009-09-15T18:17:00.004-06:00

One of my students recently asked me an ORM related question . He had this little project he was working on to present in another class, and there was a little screen where he had some data. Let say he had a table with clients (relational, you may guess), and the client had a reference to the company table, since a client belongs to a company. Of course, the reference field in database was a Company ID.

His question was very simple, although very useful to explain some concepts on the spot. He was, of course, converting that client row into an object, to be later displayed in the little window. So he asked me: “Should I add a company ID attribute to the object, or should I add the company name directly?”. I looked at him and asked him back: “How do you feel by adding that ID to the object? Is it natural?” He glanced at his computer and then said… “No.”

Then we began to discuss. Actually, we started by defining what was that object’s function in the application, what was its role, what was it there for. The answer was very simple: it was there to provide info to the screen to display the data. Then, we went into discussing what the user needs, and the answer was the user needs the company name, not an ID. Lastly we went into discussing if we needed the tow objects, one Client object with a company ID and one Company object with a matching ID and a Name. We found out that, having those two objects, will require us to match them. That is a relational operation that was natural to the database. So, it was obvious we better let the database do its job and send us a new hybrid object with the client information and the company name already in place.

But wait a minute: It that ok? Weren’t we breaking the relational rules and normalization of the data? Thinking a couple of seconds I would say no. We weren’t.

Data is data, and it is stored and related in the database. You can process and produce new information by manipulating that data using the DB tools. That new information is not to store, but to consume by the business logic layers. That is BL information, it is not simple data anymore.

So the idea of ORM one-to-one does not fit in this example. If we have data and we have tools to process that and transform it into business information, then we should allow the BL layers to consume that BL info and not to spend cycles in processing, by hand, the data. Now the objects model the reality, not the data structure, and we work with cooked information and not raw data.

William Martinez Pomares.

Reference: An ORM Story

WADL, REST and WSDL

2009-08-26T21:40:00.002-06:00

Paul kindly pointed to a blog entry about WADL use in REST, (WADL definition is found here). The Blog entry was interesting to read.

I want to quickly explain that WADL is meant to describe the possible interactions with a Web Application. Thus, it has methods, but those are HTTP commads, and it also describes the query variables (like the fields in a form) that the method needs to send to the web application. Finally, it describes the expected response.

Now, in the blog I see some misconceptions. First, it relates SOAP and WSDL to XML-RPC. That may seem true due to actual use of those two, but SOAP is just a wrapper and WSDL does not enforce XML-RPC, using document/style it reassembles a simply document sending paradigm. By the way, XML-RPC and WS used to do RPC are two completely different things.

Second, the REST based Service concept is defined to encompass even static web pages. May REST style architecture handle resources in the web, and REST “style” Services are services implemented using the REST underlying paradigm. But Static pages are not Services.

Third, WADL is at implementation level of the HTTP call, thus it does not define the service, just what to call and what to expect in response. That is not bad, it is actually good, but then WADL describes an HTTP interaction, while WSDL describes a messaging interaction. Although not actually standardized, WSDL may allow you to send messages using a message queue, SMTP, or any other transport.

Now, I want to explain my vote. I’m not a WSDL lover, I think it may be improved. Still, I’m carefull to compare, it is not as easy. We cannot say WADL is a description document for REST based Services and WSDL for RPC based ones. Right?

I may want to go even further. I may say REST Style Services may actually make use of both! Yes, while WSDL describes your Service at message level, defining the document and port, and a WADL that makes clearer the HTTP interaction (since WSDL inly indicates you are using HTTP as a transport).

Again, what do you think?

William Martinez Pomares.

Reference and other Comments: WADL, REST and WSDL

A RESTFull WSDL?

2009-08-17T19:49:00.001-06:00

In a recent comment, Paul mentions he is not familiar with the Document Style concept. This is a great opportunity to touch the theme and clarify some ideas about WSDL and the WSA idea of a service.

WSDL is meant to describe the service definition and interaction models. WSDL defines:

The Service’s visible endpoint. That is, the address where the port of the service leaves.
The Binding. That is, which transport to use and how.
The Operations. Each operation with a name and a message flow (a named message inbound or outbound).
The Messages. Each named message has a number of named parts that conform the payload.
The Parts. Each one is defined either by indicating the primitive type or a schema defined xml.
The Schema. A Types section that is defined using an XML schemas.

Using the above sections, we can define the service by indicating the port, the available operations, which message flow each operation has, and what is the structure of each message.

With the WSDL, the client is able to know how to construct the message and where to send it. So far so good for our service notion.

The problem comes when WSA came out with the style/encoding idea. SOAP was originally created to access Objects remotely (Simple Object Access Protocol). Here is the story.

Those were the times when RPC was the idea of remote access. So, SOAP was created a requirement for the body of the message. It had to be constructed with a fixed structure, where the root element’s name should be the same as the method name. The children elements should be one per parameter, and each one containing the type as an attribute. That structure was known as the encoding.

When WSA brought SOAP as the protocol to form the messages, it extended the idea in two directions. The first one was to allow some flexibility by defining a non-encoding format, and that was called “literal” (as opposite of “encoded”). The other direction is the style of the access. The RPC style is the original for SOAP, where the service is accessed by mimicking an RPC by indicating the method and parameters in the message and expect a response.
The other one is the Document Style. In this style, the idea is that the message contains a document (XML usually) that is delivered to the port.

The combination of style and encoding gives us four possibilities: RPC/Encoded (original RPC/SOAP style), RPC/Literal (A new RPC that is not that strict, see below), Document/Encoded (Nonsense), and Document/Literal (what all of us should use).

Now, let’s focus in RPC/Literal and Document/Literal. When defining the operations, the developer is able to indicate, per message, which style and encoding to use. If RPC/Literal is selected, the following rules apply:

The body of the message is an XML with a root element whose name is the name of the operation.
The children of that root element are the parameters, in order, for the implementing method.

A client calling a service’s operation defined as RPC/Literal, should create a message then that reassembles the structure of an RPC SOAP Message. The important thing here is that the XML in the body is created automatically with a root that has the operation name.

If Document/Literal is selected, the following rules apply.

The body of the message is an XML document. No other restriction is given.

Note that with document/literal, there is no restriction of the body format. If by chance the root of that XML is not the name of the operation, it doesn’t matter. In reality, some of the vendors I tested, even if document style is indicated, will look for the name of the method to call in the first element of the body.

So, now imagine a service at port http://willyloans .com, that uses http, that has an operation called LoanRequest with one inbound message. That message is a one-part message, document style. That part is an XML document defined in the schema at Types section. The client should simply create a SOAP wrapper XML, and in the body element it will add the XML document (Payload). The client then simply sends that SOAP to the address defined in service (always using a post, WSA defined that).

That’s it. Using WSDL at document style, you can interact with the service just by sending a message containing a document, delivered to a port. NO method names nor parameters. That should be a RestFull call as Paul indicates, but with the help of WSDL that describes the flow, port and the XML document schema.

Hope this explanation helps.

William Martinez.

Reference and Comments: A RESTFull WSDL?

A REAL service definition

2009-08-17T19:46:00.001-06:00

Based on several comments and articles I've read, I tend to think that SOA and Web Services are being misunderstood. Maybe I’m the one that is wrong, but just in case I will try to explain here what I understand. I guess the first thing to do is to define what a service is.

Base on the Web Service Architecture specification, a service is “… an abstract resource that represents a capability of performing tasks that represents a coherent functionality from the point of view of provider entities and requester entities.”

I want to point out the fact that a service is a “resource”, it can perform some tasks, and it represents functionality. A service has an ID that will let us find it in the networking cloud. It has a description and it has an interface. That interface “… is the abstract boundary that a service exposes. It defines the types of messages and the message exchange patterns that are involved in interacting with the service, together with any conditions implied by those messages.”

Let’s put it in other words. A Service is a coherent functionality offered by a web resource. To communicate with it, you use its interface. It will accept messages interchanged in a predefined pattern.

But there is an important fact: “To be used, a service must be realized by a concrete provider agent.”, WSA spec says. In other words, a service is an abstract model of a resource. That functionality is to be realized by a concrete agent, may it be software or hardware. Let think of it as a functional cloud, accessed by messages delivered to a port.

An Example may help.

Service: Loan service.

Functionality: Actually, the service is able to process loan requests, payments, notifies due loans and you can ask for a loan status.

How is it done?: There are four document types, each one has a particular schema: LoanRequest (input), LoanInfoRequest(input/output), LoanNotification (output) and LoanPaymentNote(input). Any input XML document may be sent to a port located in www.willyloans.com/loanport or to the email loanport at willyloans.com. There, an XML processor will determine the schema type, and send the actual data to the loan analysis department, to the loan payments subsystem, or to the loan service desk system. You can guess what each system does, and what each document may contain.
You can create four WSDL if you want. All pointing to the same address. Just make them document style.

Note that the service doesn’t reassemble an object. It has no methods or attributes, just functionality. You don’t call it (although pure objects are supposed to receive messages), and they may not have a response (although they can send a message back, or the same modified message).

Mainstream is actually building web services from the actual implementation. Instead of creating a service that is realized using exiting classes and systems, developers are simply exposing their methods and objects. They are using an RPC model instead of messaging. They are architecturing a distributed object system instead of services.
This leads to several myths and perceptions that make tool vendors offer what developers want, and none is working towards a truly service development.

I will check on those myths in future posts.

William Martinez Pomares

Reference and other comments: A REAL service definition

YAMTCARO and the Service apology

2009-08-17T19:45:00.001-06:00

I just wrote an answer in TSS that I thought I could share here too. The basic idea everyone has about services is that they are to distribute logic, make remote calls, have loose coupling interfaces, and are resusable. And nothing is working as it is advertised. I will play the defendant part here.

“I see a repeating idea about SOA, services in particular, that doesn’t fit in what I have learned from my childhood as an architect. I will explain myself.

I recall the old Object oriented days when all the fuzz was about reusability. The idea was clear: separate the code into coherent and functional complete things. Then a problem was found: the interface. You could not reuse the thing because of the interface not being suitable, or requiring things not in the new problem domain.

Then a service came. It was the idea of having a business functionality (not an interface, not an object) able to accept messages and offer a business value. It has no reusability concept, the fuzz was actually about replacing fixed interfaces with a simple port. And somebody said: Let’s implement it with SOAP. And SOAP was Simple Object Access Protocol, an RPC oriented wrapping protocol used to access distributed objects. Wow. Wait a minute. Something is not working here. I know objects were supposed to accept messages. Nobody uses that. But a service is not an object. A service has no methods or parameters. Definitively, it has no response. There is no interface per se, just a port where you deliver your message.

Now, the message is not SOAP. SOAP is a wrapper. The real thing is in the body.

Once the service has read from the port the message content, it will now know what to do. And it may decide to send another message. As simple as that. But as difficult to grasp for an object programmer, that it was reduced to a YAMTCARO (Yet another method to call a remote object).

NOw, can somebody tell me why a service needs a method name to be called? SOAP had stubs in the client, to simulate the remote object as local. Services are not object, stubs have no place here. Makes no sense. You need a port object to stream your message. Simple.

Finally, since a service is just a port, and the service semantics (which are not technically describable) is hidden behind the service cloud. You can create a versioning system underneath the cloud, so old messages can still be accepted and new messages, that represent new business needs, be processed as required. Service cloud still as white as before. It may have a new port, but old ports may be still there.

Enterprise architects? Their job is to have one eye in the problem space, and the other in the solution space. They will work with business people to design services scope (business driven), and work with designers to technically create the structure of the cloud.

Hope this helps to understand why I say all that I say.”

William Martinez Pomares

SOA in all sizes

2009-08-17T19:38:00.000-06:00

Funny article in SOA Masters Class site. It explains the vision of SOA based in two variables: the Architecture involvement scope, and the Service concept scope.
Yep, there is a big difference when you see the Service concept as an exposed component, or as business functionality. Same for involving the architecture simply as a glue to interoperate disparate systems, or as a business driven structure.

Funny indeed!

William Martinez Pomares

First Post, an SOA discussion!

2009-08-17T19:33:00.000-06:00

Hi All!

Very excited that I’m writing my very first post in the blog.
And since this is an Architecting blog, I thought I could point it to a very interesting discussion held at The Server Side.

The article presented here is about the complexity generated when using SOA, compared to the same complexity history of CORBA times. Although the idea is nice, most of the participants show a problem of understanding what SOA really is.

System Integrator Technology? CORBA with a new acronym? A new distributed approach to organize business logic? Simple EJB, SOAP and other technologies follow up, with nothing new? No several times.

As I mentioned in my responses there, SOA is based on the service Metaphor, which is placed in the problem domain, problem space, where business organization rules and not technology trends. SOA now is a patched, almost like a stovepipe architecture, solution for integration, and like that it sells.

Later I will write a little bit more about the Service metaphor. That is not for RPC developers, I’m sure.

William Martinez Pomares

The Case of Emergent Design.

2008-09-23T11:38:00.001-06:00

This one I owed it since a long time.
In the year of 2000, David Cavallo published his article "Emergent Design and learning environments: Building on indigenous knowledge". In this article, the term "Emergent Design" was coined, as an alternative of blueprints. Cavallo indicates that many analysts researching the situation of educational change projects failing, was actually because of the committee-generated blueprint that was created to drive the change. The result was always a reformed implementation of the blueprint that did not solve the problem.

So, Cavallo proposes another "way" of doing that process. He bases his discussion in two innovations: The Digital technology and the Administrative decentralization. The second one does not change the management practices, but only lowers them to other levels in organization to be distributed.

Now, Cavallo proposes some principles to guide this. Constructionism (Papert, we build knowledge when we construct things), that is based on Constructivism (Piaget, we build upon what we already know). The other ones are Technological Fluency, Applied epistemological anthropology, immersive environments, critical inquiry, long-term projects and Emergent Design.

The idea was to allow children to be immersed in long period projects, so they can handle fluent technological language. With that, the students will start looking with criticism the new understood world, and using the epistemological anthropology the teacher may be able to detect the learning forces of the students freedom and design "on-the-fly" the next learning framework to explode the possibilities.

All this is handled using the emergent design approach. The learner focus on building new knowledge upon the existing knowledge, and the tools and guides are given to facilitate that. Thus, there is no preexisting path, but one that is built as it goes on. It is pure Discovering by Construction.

Now, that is a learning experience Cavallo says it is not the only one possible. It was mapped to software development by Kent Beck, and it is presented as an alternative to blueprints (that is, the dreaded BDUP).
Emergent design is built on top of a Make-Verify-Improve life cycle. Since the new world is not created yet, the developers usually give baby steps, one step at a time. Decisions are not taken at earlier stages, but a just-in-time approach is followed, and developers are the one making those decisions.

These can prove good points:

Avoids Prediction. Decisions are taken upon knowledge acquired by constructing, when the developers know that it works, and at the time it is needed.
Verifiable products. The created products are real, working, and thus they can be validated against the requirements. New development is done on top of this real thing and not on top of paper ideas.
Decisions are backed up by information obtained from feedback of real products and not assumptions.

Still, there are a couple of bad points to mention:

Refactoring vrs Redesign. Note these are two different things. Refactoring is about improving the code after it works, redesign includes changing what it is done to use another approach or structure simply because the first one does not work. So, emerging design may not be about creating and then improving, but creating and then recreating.
Late decision at Macro level. The problem with too late decisions are that the ones that affect a level higher that code may require much more amount of rework, recreating functionality in another way. Remember, refactoring is work to improve, rework is work to make it work as it should.
There is certainly more information (a developer immersed knows more about his local environment than a designer), but there is less higher view to make the correct decision. That is, the grass-level view Cavallo explains work for individuals creating a single knowledge, since that knowledge is ok to be incomplete (there are lots of data the learner will not know). When you have individuals working together, you need a second level view to understand how all the individuals are doing matches together.
The validation is locally centered. When individuals validate their decisions taken upon the current local value, the isolated solution may work. When integrated, the correct parts may not add up to an integral sound structure.
It is assumed the individual will make the correct decisions based on information, which requires a long time to acquire (Cavallo’s long process). Thus, initial development is a challenge since no individual has enough info to make certain decisions.

So, those and other pitfalls can be avoided by taking care of tactical and strategic decisions:

Use guidelines, like principles, to driven the decisions in case of doubt.
Create from the bones. That is, create on top of a secured and sound base that explains by itself the rationale of the solution. That may require to bring some critical decisions Up Front, always knowing those decision may be changed in the future.
All decision has to have a rationale that supports it. The idea is that later on, when the decision proves to be wrong, the rationale may help to chose another idea.
Multilevel Coding. I know this is something not much people understand, but actually defining the global structure, then defining each component using a tactical structure to bind the bones, and finally writing the code to add meat to them, is what I call coding at levels, since every description of the solution is code. This helps a lot to visualize when and how a decision of any level can be taken or improved.

Hope this helps to clarify a myth about Emergent Design: It is not an easy two step methodology of blindly doing and then step back to see what emerged from the disconnected decisions. And, it may not be good for all projects nor for all people. Lastly, it is not an alternative to architecture, but just another technique to construct the architecture itself.

William Martinez

About the NOOB definition– Part 2: Globalization.

2008-04-26T13:50:00.001-06:00

Prior the Internet days (some people talk about prior television), we had different villages, local ones, with no communication. Now, we live in a global village. We need to communicate and share knowledge, even in different languages and cultures. Furthermore, we learned it is not just us and the "my limited world", but any thing we do affects the globality.

Now, when working on a real world problem, working to find a solution of course, we need to communicate and all or our work is to complement the one from others. There is no one single person that fights all the villains and the rest of the team observes and cheers.

Solving a problem with a software intensive system, involves working with all parts of the system: Hardware, Software, people and processes. No one lives alone with its computer, everybody interacts with all the other components of the system, and the developers are part of that system too!.

Making sense of that, the strategy of working only in my local line forgetting there is something beyond that, goes back to the local world and denies globality. Does it mean you need to work all at the same time? No. It means you work in your local line knowing there is something bigger of which that line is an important part.
What does this matter in Steve Yegge’s post about NOOBs? Well, going extreme, the post indicates there is much waste, fat we need to remove to be healthy. The comments, even the modeling and describing language structures are needless for the seasoned, non-noob programmer. I agree, as long as the programmer is working in a system that has only two components: the programmer and his lines of code.

Unfortunately, that is not the reality. A system not only requires the work of many other components, some non-software and some non-programmers, but also requires all those components to communicate at design and execution time. Robert Martin once said that one of the software functions is actually communicate with people, readers that may have not participated in the development. Does that mean the comments should stay? Not exactly, but that the code itself should be readable, understandable.

One way of doing that, for the development group, is using sentences that make sense and do the function (you can scramble the sentences and do the same thing but being not readable).

Globalization in the System means all components will need to communicate and work with other components. Creating the product with those characteristics, because they are needed, is not being NOOB, but being aware of the fact the software is not an island.

William Martinez.

About the NOOB definition - Part 1

2008-03-30T13:46:00.001-06:00

Paul pointed me to a TSS post I missed. Steve Yegge posted some rants about complexity and got to a conclusion about the how good and clean are scripting languages (Portrait of a NOOB ).
I couldn’t post a comment to it, since it was already closed to comments. So I guess I will answer here.
In this first part of a response to that blog, I will reproduce the post I did to TSS. Probably nobody read it since I posted it one month later.
First, let’s summarize Steve’s post (or what I understood of it):
1. Comments are metadata. Metadata is of no use for the compiler, so it can be get rid of.
2. Noobs are comment obsessed creatures, thus metadata enthusiastics.
3. Modeling is a way to comment intention of code in code language itself. So modeling a metadata creation process.
4. Teenagers are obsessive modelers.
5. Classes and in general OO, are metadata in code itself.
6. Static typing is also metadata since the real thing is memory afterwards.
7. Seasoned Programmer is one enlighten creature that does not use metadata, and thus can see the real thing when coding, not ethereal domain models.
What do I agree with:
Metadata is useless for the compiler. True.
Modeling is a metadata creation process. True.
What do I disagree with:
Metadata (at least not all) is not useless. Metadata is important, although not for the compiler.
Probably one of the things Steve Yegge misses in the post is that a language has much more than just telling the compiler what to do. In the DSL post, we talk about languages oriented to one specific domain, using vocabulary and grammar of that domain only. The idea is languages are there to help you solve a real life problem, not to help the compiler. Languages are for you (and your peers), not for the machine. And maybe not all languages are meant for a computer to read them.
Furthermore: Lack of metadata does not mean better code.
Here we talk about two very different issues: Code quality and the actual coding process. One may affect the other. A good coding process should help produce healthy and good code quality.
Any code with or without the comments will execute the same, good or bad. So comments may not add quality per se.
But, adding comment may affect (and improve) the coding process.

Now, here are my definitions:
Coding: Process of describing the solution using a specific language or set of languages.
Language: Set of words (verbs, nouns) and grammar, used to create describing sentences.
Coding Level: Abstraction level at which the solution is described.
Compiler: Entity that will convert a solution description down into another description in another language (usually a more detailed one).
So, for all of you that think that your Java lines of code actually execute, sorry to tell you that is not true. Those lines will be converted into bytecode. And that is another language that will not execute either. It will be converted again into (probably C) calls to the Operating system at hand. And those to assembler codes, to binary codes and then to electric impulses. Lots of compilers, huh?
Well, for each compiler of Level X(n), the language of compiler X(n-1) is a level higher, and contains sentences that entities of that level can understand. And for X(n) those are just a less detailed solution. Since the sentences are actually the same solution, I may say X(n-1) is just a modeled solution for X(n-1) entities, and thus metadata for X(n).
So, metadata is not a useless waste of characters, it is simply of use for other entities that are not the compiler at hand (Junit 4 and TestNG are proof of that). If you fail to see that, is because you are looking at your compiler level, with your actual language rules.
Thus, for me, a NOOb is when you go out the street and start walking, knowing your block (you talk too much about all that is in your block). Teenage is when you have a car and star driving to know your city (you talk about love and other things, not much about your block). Seasoned programmer is when, using a SUV, you start climbing mountains and going out to wild lands and foreign countries (you start talking other languages). You become an analyst when you drop your car and start studying the world by TV and news channels, and fly around in helicopters and planes (you now talk politics, business). You end up being a senior ancient, when you take your ticket to the space station, and see the whole world at a glance, and start talking about future.
Will

TDD or not TDD?

2008-03-23T18:19:00.000-06:00

In a presentation by Cedric Beust and Alexandru Popescu at QCon San Francisco, about "Designing for Testability", called "Next Generation Testing", and in his post, Cedric talks about "Test Driven Development" (TDD), and does not talk to promote it.

Actually, he does not talk against it, but simply questions about using it, as if it is a good idea or apply-able to all cases. Cedrics points several problems of using Test-Up-Front technique, and one of his lines is: "Promotes micro-design over macro-design". Follow me on this...

Let's change the last D from Development to Design. A Test Driven Design is that one that is accomplished by basing the decisions on the test (The principle may read: follow the option that is best testable). No one is saying when to test, nor if you create the test before or after the actual coding. We are actually thinking on design, and taking into account one -ility: the Testing one.

Now, from my post about the design levels, (and here is what drove my attention over that Cedric's line), the term "micro-design" is defined as the design decisions the developer makes during the implementation of the "tactical design". Those design decisions should be in accordance to the tactical design, and those should support the strategic design.

Here, the question is now if the Test Drive Design is driving a design level that is not the micro design! Tests, if we talk about Unit and Functional tests as I think TDD means them, are a developers level thing. They, in this case, should drive Micro-Design only, and should never drive other design levels.

Let me explain it in other words: TDD shouldn't tell you what classes to use, which artifacts (queues, files, etc) or which relations between functional units there are. But it may help you decide which call sequence to use, which functions should a class have, which structures to rely on. Those are micro-design decisions, that may later on be refactored.

TDD, used at micro-design level will help define interfaces, improve cohesion and even enforce encapsulation and information hiding (if used correctly, of course, since we can create tests that violates all those best practices). But that is when the developer is creating the operational level descriptions (code). There should be a tactical design in place before doing this. And there should be an strategic design in place before that. We can use the testability to drive some decisions at those levels, but they cannot be the main -ility at all.

Design Levels

2008-03-16T18:15:00.001-06:00

Working with developers, I often found them thinking of design as a generic word to indicate "sketch the classes you are going to create".

Of course, I don't share that definition. I think of design as:

"composing the sentences, in a particular design language, that describe the solution of the problem"

That said, I add that there are several design levels and types. That is, you may find out you are designing when you think you are doing something else!

Amnon H Eden and Raymond Turner had some publications where you can read about a proposal for an ontology (using the Intention/Locality). Part of their definition is that the software design should be divided into three categories: Strategic, Tactical and Implementation.

Of course, Strategic design sentences are those for Architectural design, where decisions will impact the whole. Tactical sentences are what we have always call the design, where the focus is local to one part of the problem, and that is abstract enough to be variable. Then the Implementation are sentences that are local, but also the actual execution.

That last one I call Operational design. I also call it Micro Design. That design is the one the developer does when creating the sentences that define the executable solution. Note that the decisions here are also important. Note also there are design patterns to this level too, that are different depending on then selected language (those are called Idioms).

There is another point to this definition of design: We can describe a solution at different levels, and it will be the same solution. The good point here is that each level is an abstraction of the lower level, where details was taken out of the way to clearly see broader requirements. Not all designs can be executed, and not all levels may have all the required details to be able to execute as it should.

Funny, huh?