Service Oriented Architecture (SOA) Design

Service Oriented Architecture (SOA) Design Introduction Objective The goal of a Service Oriented Architecture (SOA) is to achieve reusable assets in an agile manner that is aligned with the business needs of the organization. The SOA Reference Architecture provides guidelines for making architectural and implementation decisions. To that end it serves as a blueprint for creating or evaluating a solution architecture for different groups within the organization. In addition it provides insight and patterns for integrating the different elements of an SOA as illustrated by the different layers of an SOA. An SOA Reference Architecture is designed to answer some of the key questions and issues encountered while developing a solution such as: What are the aspects of an SOA as expressed in terms of layers that need to be considered when designing solutions based on SOA principles? What are the building blocks needed to include in each layer of a solution? What are some of the key architectural decisions that need to be made when designing a a solution that is based on SOA? Which roles in a project would benefit from using these principles and guidelines? To answer these questions this document provides the following: Establishes a common vocabulary and set of definitions for services and SOA. Identifies the different functions of an SOA implementation and define their interactions with each other and with functions outside the scope of the Rufus platform. Provides SOA guiding principles. Defines consistent design and implementation across services. Shows when and where to use SOA technologies. Supports SOA governance. Provides an architectural based design methodology. Overview This document presents the layers, building blocks, architectural and design decisions, patterns, options and the separation of concerns needed to design or evaluate an SOA architecture. The architecture consists of multiple components which are divided into the following categories: Quality of Service (QoS) Security Consumer Business Process Integration Services Information Infrastructure Governance Monitoring Management Center of Excellence (COE) Design Time Elements Component View Design Time Elements Design-Time Activities are the technical activities, guidelines, and deliverables that allow for everyday development within the SOA and Integration Architecture. Including: Service Modeling and Design Canonical Data Modeling Policy Specification Service Assembly Service Testing Service Identification Service Lifecycle Management Service Modeling and Design Services must be modeled and designed to support an extensible plug-and-play model by being: Reusable So that other Services leverage an existing Service. Composable So that a Service can leverage other Services. Interoperable So that Services can work with each other. Discoverable To support location independence. To enable the above qualities, each Service (except for Data and Utility Services) needs to have: A well-defined Service Contract To enable loose couple by hiding low-level implementation details from Service Consumers. SOAP-based Web Services achieve this by specifying a WSDL interface with XML-based document exchange. RESTful Web Services define a contract with HTTP input parameters and JSON/XML-based responses. Coarse-grained So that operations have enough data to carry out all tasks in a stateless manner. Stateless operations Statelessness (i.e., not maintaining state between invocations) enables a Service to be reused in many contexts. A way to handle Exceptions to notify its caller if there are problems carrying out an operation. To ensure that each type of Shared Service is designed and implemented in a uniform manner, a SOA Governance team should create SOA Best Practices documents for: Service Design, Web Service Design, and XML Message Design in Designers Guides (i.e., working agreements). Topics would include: Service Specification and Design Interface (including Canonical Data model for messages see section 3.5.2) Policies (see section 3.5.3) SLAs (Service Level Agreements) A description of what the Service does A description of the Services operations. Best practices for each type of Shared Service (Business, Common, Data, Integration, and Utility). Examples showing the difference between Service-Oriented Design and Object Oriented Design. How to develop a SOAP Web Service (and handle Exceptions). How to develop a RESTful Web Service (and handle Exceptions). Canonical Data Modeling (XML) Many organizations have several autonomous packaged and custom developments that have evolved independently of one another. Many times, each system has become a system with its own implementation methodology, culture, processes, business rules, and vocabulary. Many companies face the problem of using the business data to enable applications to communicate in a distributed systems environment. Organizations design Canonical Data Models (CDM) to help analyze the message exchange within the organization and with their trading partners. Data is embedded in the basic architecture of any organization. To develop the CDM, an organization must inventory business concepts and map the vocabulary into fundamental business concepts. The CDM provides a framework for integrating the disparate terms for each line of business. The CDM is a catalog of neutral terms defined in an XML Schema, including: Entities in the business domain (e.g., Order, Customer, etc.) Agreed-upon data structures so that a domain element has a single, common definition. Formatting rules. A CDM defines an organizations data in motion the XML messages exchanged between: Services within an enterprise. An enterprise and its external trading partners. Many industries have an industry standard (e.g., ACORD, for Insurance, or EDRM for e-Discovery) to define core concepts and define data exchanged between companies. A Canonical Data Model provides the following benefits: An enterprise can easily exchange data with its business partners. An enterprise can adopt a single approach to exchanging data across all services the enterprise. The canonical model helps define the interface for each Service (thus hiding the structure of the physical data in the database). Changes to the canonical data model are required only when an entirely new business concept arises. Policy Specification Policies provide service-level meta-data to Web Services consumers. Specifying these policies is an important step in Web Service design, and this is accomplished by attaching policy expressions to the WSDL. Specifically, these policies include: Security: Transport Protocol Algorithm (RSA, etc.). Messages Authentication/Authorization Level of Service (e.g., Silver, Gold, Platinum, etc.). For example, a Platinum customers requests execute on best-provisioned server for better performance (but also at a higher price). Performance For example, send notifications if performance for a particular service dips below the promised SLA. Auditing For example, begin auditing when a particular service consumer invokes an operation on a Service. Service Assembly A Business Analyst first develops a Business Process Diagram (using BPMN Business Process Modeling Notation) to model a business process at a business/requirements level. Then, designers and developers create a runtime model in either BPEL (Business Process Execution Language) or JBI (Java Business Integration) to: Assemble existing Shared Services into a Business Process Enable a BPMS product and/or ESB can run the business process. Designers use MEPs (Message-Exchange Patterns) to model the runtime characteristics of a business process. Typical MEPs include: Pipes and Filters Content-based Router Recipient List Wire Tap Dynamic Router These patterns are gaining in popularity and many of the newer ESB and BPMS products support executable MEP models. Please see the following for further information: Enterprise Integration Patterns Home Page Enterprise Integration Patterns, Gregor Hohpe, Bobby Woolf, et al. Service Testing Most Shared Services will be implemented as Web Services, and it is important to take a Test-Driven Development (TDD) approach to Web Services development because: Designers and developers need early feedback on the design, functionality, usability, and performance of each Web Service. QA personnel need to be able to test Web Services. Overall quality is very important because each Web Service could be used by multiple consumers. Service Testing includes: Inspecting Web Services: Documentation Generating HTML documentation from the Web Service interface. Debugging Show SOAP/HTTP requests that are sent received over the wire. Invoking Web Services: Generating Web Services requests from WSDL to set the Web Service. Simulating/Mocking Web Services Generating Unit Tests based on the Web Service interface. Functional/Performance Testing Web Services Simulating load conditions. Reporting on performance under load conditions. Commercial products include: Eviware soapUI Pro iTKO LISA Open Source products include: Eviware soapUI Community Edition WebInject PushToTest TestMaker Points to Ponder (for Evaluation) Does the product support both SOAP and RESTful Web Services? How? Does the product enable you to view SOAP/HTTP messages over the wire? Does the product have a good/usable Web UI to make it easy to test? Can a QA/Tester use the product without developer intervention? What level of SOAP, WSDL, and HTTP does the product support? Does the product support an Agile/TDD approach with Unit Tests and Mocking? Does the product generate Unit Tests to support Continuous Integration? Can the product simulate load conditions and report on performance/scalability? How much setup is involved? How does the licensing work? Service Identification Service Identification is one of the key steps in designing a Service-Oriented solution because it defines and identifies high-level Business Services by using the following the following approaches: Top Down: Analyze and model business processes. Design new Services that enable the tasks and activities in the business processes. Bottom Up: Service-enable existing and applications systems. Create business processes from the Services. Middle Out: Analyze and model business processes. Catalog existing applications and services. Determine which can be Service-enabled. Create Service Adapters. Map tasks and activities from the business processes to existing Services. Create new Services to fill in the gaps. A Middle-out approach is recommend because it takes the best parts of the other approaches: The bottom-up approach creates isolated silos that dont align with the business. The top-down approach takes a long time, and the enterprise cant wait for every business process to be defined in order to begin designing and developing Services. A compromise approach takes into account the need for timeliness, but also instills enough discipline to design Business Services that match up with the goals and objectives from the Business Architecture by: Taking a first cut at the Business Services from Marketing materials and meetings with the CMO and other business stakeholders. Starting with only a few Business Processes. Driving toward a thin/vertical slice of functionality based on the business processes. Service Lifecycle Management Service implementations are software like any other software module or application. As such, they go through a similar lifecycle. The service lifecycle is depicted in the following diagram. The lifecycle starts with Service Identification. Services are part of more general business processes. As such, new services are usually identified by the design of a new business process. This is not the only way that new services are identified however. Sometimes services are identified as part of portfolio management. This involves analyzing the requirements for a service across the breadth of the enterprise. The level of analysis required for this type of identification is difficult to gauge and therefore CIBER does not recommend using portfolio management for service discovery at this time. Once a service has been identified, it follows an iterative development process. Iterative processes use the feedback from subsequent phases to make corrections in previous phases based on lesson learned or issues that may crop up. Services are part of a greater distributed system however and any changes made can have an impact on other development efforts. Therefore it is important to be mindful of the effect change can have even during development of a service. The Service Specification and Design phase produces at a minimum the specification for the service interface. This interface specification includes the semantics and data that the service supports. This represents the contract between the service and its consumers. Diligence should be applied to the design of this interface as changes to the interface have the greatest impact to subsequent phases. Changes to the service interface can impact both clients that may be developed in parallel, implementation of the service, as well as test plans that have been implemented to test the service. This does not mean that a service interface must never be changed once it is designed. Designs are not perfect and in todays IT environment it is not always possible to take the time required to produce an interface definition that is ideal. Therefore it is prudent to put processes in place that take into account that service interfaces may change. Whenever implementing a service take into account tha t the interface may change to the extent possible to minimize the impact of that change. This also applies to the implementers of consumers of the service. In some cases, consumers may want to wait until the service has been through some number of rounds of testing before starting their implementation. This allows for some experience to be developed with using the service thereby (hopefully) minimizing the possibility of change to the interface. Once the service has been designed it proceeds to the Service Implementation phase. In this phase the service will be developed based upon architectural standards developed by the university OIT group. Any issues encountered trying to implement the design of the service should be fed back to the designer in order to refine the design if necessary. Finally, service need to be tested before being deployed into production use. Testing of services involves four primary areas of focus: Security testing is essential to assess the risk of a service with regard to vulnerability, data privacy and data integrity. Tests need to be developed to test boundary conditions which can assess the robustness of the service handling inputs outside the range of anticipated values. Tests should also be created that ensure the service performs as expected based on the roles as defined within the system. Type of Testing Description Functional This area of testing focuses on ensuring the service performs its function according to the requirement of the business process it support. Automated test suites should be developed to perform regression testing to quickly verify functionality during the life cycle changes that may occur. Performance This area of testing focuses on performance characteristics of the service including measurement of time to perform the service and load testing of the service. The output from this type of testing forms the basis of understanding how to configure and deploy the service in a production environment. Interoperability This area of testing ensures the service adheres to its service specification. Early identification of interoperability issues is key to integration of the service through exposure to university partners and clients. This type of testing is especially important when the service interacts with multiple data sources and/or systems. Security Security testing is essential to assess the risk of a service with regard to vulnerability, data privacy and data integrity. Tests need to be developed to test boundary conditions which can assess the robustness of the service handling inputs outside the range of anticipated values. Tests should also be created that ensure the service performs as expected based on the roles as defined within the system. Regression One of the more important types of testing related to usage of services is regression testing. The more applications that depend on a given service, the more impact a change to that service can have on the environment. As such, when changes are made to services, regression testing must be undertaken to ensure that the service not only supports new or updated functions but all other functions upon which the service relies. Table 1 Service Testing Types Once the service is ready it is deployed into production use. At this point the service enters the maintenance cycle as opposed to the development cycle. Services in production sometimes require change. The change may be due to a defect in the system or a request to add more functionality to the service. It is important to understand how this change should be handled based on its nature. Defects that are a result of a fault in the underlying implementation of the service and do not change the behavior of the service may be handled by a Defect Remediation process. In this case it is desirable to apply a fix as quickly as possible to the existing service as it may be affecting multiple university processes and causing a disruption to the universities ability to perform its function. Changes that modify the behavior of an existing service or its interface are best handled by identifying a new service or a separate version of the service. In this scenario versions are really new services that are separate from the original. Taking this route minimizes impact on consumers using the original service but can cause a proliferation of services within the enterprise. This may not always be desirable however and some effort should be put into defining conditions under which an existing service in production may be modified (for instance, only one business consumer is affected and the change is well understood). This should be documented so that everyone understands the conditions and ensure that changes are handled in a consistent manner. Service Withdrawal is the final step of the lifecycle. Eventually services will start ballooning (especially when changing production services leads to the creation of new services) and some services will stop being used. Removing service can be problematic as a service may support multiple business process owned by different colleges and departments. A procedure needs to be adopted to define and orderly withdrawal of services from the system. Such a procedure may start with deprecating the service (with a note explaining why and some suggestion as to another service to replace it). Secondly, services can be monitored for use and the consumers identified. Finally, if the service is still being used the consumers should be contacted to discuss a solution. This should result in establishing a schedule for the consumers to switch over to another service so that the service may be withdrawn. Consumers of a deprecated service may not have incentive to change. Making changes requires effort and possibly some risk to the consumer. This needs to be recognized when seeking collaboration from consumers to switch over to another service. In this case it will be necessary to be creative in coming up with a common understanding of the benefit of switching to a newer service so the deprecated one may be withdrawn. Center of Excellence (COE) An online community to: Impart the SOA Vision. Educate IT and business staff on SOA. Communicate SOA best practices. Gain feedback on how to adapt the SOA Governance process and overall SOA program. Provide support advice for new and ongoing SOA implementations. Provide SOA Resources: Books Web Sites Industry Standards Quality of Service Logging Most applications and systems use some form of logging that stores messages to a persistent medium (DBMS, file, etc.). Logging provides the following benefits: Troubleshooting Applications log errors upon failure, and system personnel use this information to repair the problem. Reviewing System personnel examine log messages to check for problems. Auditing Security personnel can review log messages to see what actions a user performed in the system. Debugging Developers generate log messages to debug their programs. Identity Management Identity Management is the ability to identify a requestors (person or system) true identity and relationships between people and organizations (groups). The current policies and level of enforcement are expected to continue into Managed Services by using Microsofts Active Directory as the source for identity management. Confidentiality Confidentiality assures that during transport of the data it was never visible, accessible or viewed other than by authorized recipients. There currently is no Smart Energy or Smart Grid requirement to secure messages for Confidentiality. Authentication Authentication is also done within Microsofts Active Directory and adds authorization policies to the verified identity. Authorization Authorization is currently done at the application level. Within Managed Services (stage MS3) authorization will be performed to determine whether the requestor (person, application, service) is authorized to access the requested service, data and even the Managed Data Repository. Integrity / Non-repudiation Integrity requires that during transport and even as read by the sender there were no unauthorized modifications of the content of the message. Non-repudiation assures the sending service that the receiving service has received the intended message. Current Smart Energy and Smart Grid projected needs do not require message Integrity, but for sensitive messages (time, confidentiality, event, or priority) there is Non-repudiation. Consumers Service Consumers are the end consumer/user of the services provided by an enterprise. The consumer has the flexibility to process and display useful and relevant information provided by Services. Service Consumers access Services through a consistent interface (or contract) exposed by that Service. Service Consumers can be: User Interfaces B2B Applications Business Processes User Interaction Users can access enterprise services through variety of mechanisms, including Portals, web sites, or PDAs. A web site provides a web-based interface to enable users to perform daily, job-related tasks. A web site leverage the services created provided the organization, as a part of SOA implementation, and do not contain any business logic themselves. The site seamlessly integrates with back-end services (using SOAP/WSDL, REST/JSON) and business processes. The site can be secured by global security policies, but also can include role based authentication that limits access to only relevant information making it easier to manage. A Portal is a web site that enables users to access highly personalized information and services. It can increase the productivity and effectiveness of employees within an organization through a consolidated view of available services and information. Typically the site is highly interactive, allowing the user to run a wide variety of tools and functions such as, global search queries, and custom dashboard and advanced business intelligence tools. Portals provide a unified entry point to the organization and provide common look and feel all across applications. Portals form the front end for business processes and custom applications created as composite applications. The site can also be utilized to mash-up other applications or services from 3rd party sites. Wikis, Blogs, RSS feeds, and content can all be made available within the site. A PDA enables mobile users to access enterprise services. Like sites and portals, a PDA has no business logic of its own, but it allows the end user to interact with back-end services by seamlessly accessing the Web Services exposed by an enterprise. However, a PDAs interface and functionality is much more limited than that of a web site because of display and memory constraints. B2B Applications An organization normally collaborates with external business partners such as suppliers and customers to achieve its business objectives. An external partners B2B application (e.g., a web site or portal) will invoke a Web Service exposed by an enterprise, which in turn executes business functionality on behalf of the client. Business Processes A Business Process codifies and streamlines the rules, practices, and business s activities in an enterprise. Business analysts create Business Process Diagrams using the industry-standard Business Process Modeling Notation (BPMN) to document a set of coordinated tasks and activities that lead to an organizational goal. You can think of a business process as a graphical representation of a Use Case (RUP) or User Epic (Agile / Scrum) in that it shows normal and alternate flows along with exceptions encountered during processing. In SOA, a business process coordinates the business services (see section 3.2.1) developed by an enterprise. SOA architects and developers derive business services from the tasks and decision points in a business process diagram. Business logic is used to form business flows as parallel tasks or sequential tasks based on business rules, policies, and other business requirements. Examples of Business Processes include: Purchasing a product Time Entry / Approval Billing Service Provisioning For information on tools and products, please the Workflow / Orchestration section (3.4.5). Collaboration Services Presentation Services Presentation Services define a common set of services to manage interaction with users or trading partners (to the extent this second interaction is needed). Presentation services are provided by: web servers, portal servers, and application servers that provide the capability to quickly create the front end of business processes and composite applications to respond to changes in user needs through channels, portals, rich clients, and other mechanisms. Presentation services integrate with other foundational services, such as security (e.g., single sign-on). Users can access enterprise services through variety of mechanisms, including Portals, web sites, or PDAs. A web site provides a web-based interface to enable users to perform daily, job-related tasks. A web site leverage the services created provided the organization, as a part of SOA implementation, and do not contain any business logic themselves. The site seamlessly integrates with back-end services (using SOAP/WSDL, REST/JSON) and business processes. The site can be secured by global security policies, but also can include role based authentication that limits access to only relevant information making it easier to manage. A Portal is a web site that enables users to access highly personalized information and services. It can increase the productivity and effectiveness of employees within an organization through a consolidated view of available services and information. Typically the site is highly interactive, allowing the user to run a wide variety of tools and functions such as, global search queries, and custom dashboard and advanced business intelligence tools. Portals provide a unified entry point to the organization and provide common look and feel all across applications. Portals form the front end for business processes and custom applications created as composite applications. The site can also be utilized to mash-up other applications or services from 3rd party sites. Wikis, Blogs, RSS feeds, and content can all be made available within the site. A PDA enables mobile users to access enterprise services. Like sites and portals, a PDA has no business logic of its own, but it allows the end user to interact with back-end services by seamlessly accessing the Web Services exposed by an enterprise. However, a PDAs interface and functionality is much more limited than that of a web site because of display and memory constraints. BI / Reporting BI (Business Intelligence) / Reporting provides a high-level view of KPIs (Key Performance Indicators) to business stakeholders to enable them to make decisions and manage the business. Examples of KPIs include: Efficiency of business processes. Job Costing. New customers acquired. Sales information by demographic (age, ethnicity, geographic region, etc.). Churn / turnover of accounts by demographic. A BI product uses the information stored in a Data Warehouse to present it to the user. A BI tool (such as Business Objects) uses Key Performance Indicators (KPIs e.g., sales conversion rate, in-force polices, market penetration, for example) to report on data, identify trends, perform data analysis, etc. to enable business users to make decisions and operate the business as efficiently as possible while advancing a business strategy. BI increases business agility and shortens timeframes for decision-making. It gives companies the ability to identify and anticipate opportunities represented by seemingly unrelated events. It is a key enabler of strategic and tactical decision making. Commercial products include: IBM COGNOS MicroStrategy O