Wednesday October 24 2001

Using UDDI as a Search Engine

Smart Web Crawlers For All

Bilal Siddiqui

Printer-friendly HTML version
Or purchase the extended PDF version of this article from our download site, or from Amazon.com.

Universal Description, Discovery and Integration (UDDI) registries provide a mechanism to browse through and discover Web Services and interact with them in a meaningful way (visit UDDI.org, the birthplace and official web site of UDDI). In order to explain this concept, we will first discuss its usage model (operation of Smart Web Crawlers) and then discuss its internal architecture. Towards the end of this article, we will explain the process of searching through a UDDI registry.

UDDI registry as a Smart Web Crawler

How does a search engine like Yahoo or MSN work? The user provides keywords and the search service looks for those keywords in the meta-data of web sites that it has listed in its database. Search engines may also provide advanced features like searching within search results, excluding records that contain certain keywords, etc. No matter how advanced the search is, though, it will always depend on keywords.

Future search engines (let's call them smart web crawlers) will be better than this. They will not only search for keywords, but also find the capabilities of web sites. What if you want to find hotels in a city that fall within a particular range of per night charge? A conventional search engine can get a list of URLs for you to explore and visit each of those hotel sites to check the costs. With smart web crawlers, you can directly look for hotel Web Services that expose their functionality for giving automated quotations and reservations. UDDI enabled smart web crawlers will use HTTP, SOAP, and WSDL to get a list of hotels, and check which of those hotels publish their tariff and allow automatic reservations. You would then invoke the services of those sites to check their charges, and possibly make reservations.

Web Crawlers working for Ordinary Internet users as well as for B2B Portals

Smart web crawlers offer interesting prospects to ordinary Internet users as well as e-commerce owners. Let us take these scenarios one by one:

1. If you are an ordinary Internet user surfing the web using a browser program (maybe a PC based browser, or a micro browser inside a hand held device), smart web crawlers have value-added services to offer you. They allow you to surf through categorizations and classifications, finding Web Services that belong to a particular industry and explore their interfaces to check whether they offer services of interest to you. In order for UDDI operators to serve all these purposes, they need very user-friendly and smart graphical user interfaces that allow ordinary Internet users (who know almost nothing about Web Services) to take full advantage of this new technology. In order to discuss the design of such smart user interfaces, a detailed overview of the architecture of UDDI registries is important; we will return to UDDI architecture shortly.
2. What if you are a Business to Business Web Service owner running an on-line tourist information and e-commerce show, where finding hotels is part of your business process? Using conventional web search methods, you will have to manually find hotels, talk to them, and design a content management solution that suits both your partner's and your own database systems. Imagine you have few hundred hotels as partners, each of them having their own back-end data format (which means they are using different databases and application architectures). In this case, you would need to implement coordination and electronic data exchange mechanisms with each individual partner hotel. With smart web crawlers, the entire operation becomes smooth and automated as UDDI allows you to discover Web Services. This, together with WSDL (Web Services Description Language) and Fingerprints (to be explained later in the article) allows Internet users and B2B owners to meaningfully interact with the services that their partners offer. Therefore, whenever a new hotel enters the UDDI registry that you work with, it will automatically start appearing on your tourist B2B site, independent of their web site's application architecture.

In essence, you can think of UDDI registries as conventional search engines with the additional capability of acting as service brokers, thus introducing businesses to their prospective clients (whether individual Internet users or other web applications).

So far, we've looked at the usage model of UDDI registries. Now let's move on to the technological and architectural view of UDDI.

What's inside a UDDI Registry?

UDDI defines a fairly complex data structure that contains information necessary to discover services and interact with them. The following diagram shows a single UDDI record as a box that contains a hierarchy of smaller boxes. The complete UDDI registry will contain millions of such records.

We have labeled the largest box (solid blue) as a UDDI Record, which contains a Business Entity. The Business Entity data is made up of several fields, such as Name, Description, and Contacts. We have simplified the figure slightly and not shown the complete UDDI record hierarchy. In the actual UDDI specification, there are still smaller boxes (for example, Contacts is a list of Contact structures).

Among the other fields within a Business Entity is its Business Services list, which is a list of Business Service structures each of which represents the actual functionality of a web-based service that a particular business exposes (for example rent and on-line reservation services). A Business Service structure contains one or more (1..n) Binding Templates, which are templates that bind Web Services with their actual implementations. We can bind each service with any number of templates. A template must, therefore, contain two very important pieces of information.

Firstly, an address or the location of the place where we can find the actual implementation, and secondly some description about its architecture. In the diagram, the Access Point field provides the first bit of information, by directly pointing to the URL of your prospective business partner. Each Binding Template contains a list of tModelInstanceInfo structures (this list is called tModelInstanceDetails).

Each tModelInstanceInfo structure inside a Binding Template refers to a tModel, which is the second bit of information that a Binding Template must contain and is the most interesting part of the UDDI data structure. tModels are technical specifications describing the nature of a business. UDDI has adopted an extensible framework for describing the nature of businesses. In the next few paragraphs, we will describe tModels as fingerprints. For now, look at the Access Point field of the Binding Template structure, which can directly point to the URL of your prospective business partner.

How to Learn about the Businesses we find in a UDDI Registry

Once you have reached the URL of your business partner, there are three possibilities:

1. You want to learn more about your partner manually through conventional browsing
2. You want to discover through UDDI, but you do not know anything about your partner's Web Service
3. You know that your partner belongs to a particular community (for example, if you were looking for a partner hotel, we can guess that your search results refer to hotel Web Services)

The second option here presents an interesting case, where the next task is to somehow learn about the Web Services of a business that you know nothing about. Therefore, you need some type of mechanism that defines or describes Web Services. The UDDI specification does not tell you how to do this; instead, it allows other technologies to cater for this issue. For the time being, the favored technology for this purpose is Web Services Description Language (WSDL). The UDDI architecture is open and flexible, so that something entirely different from WSDL can be designed and implemented independently from UDDI and will have no problem coordinating with clients.

This may seem like an odd thing, but it is not only useful, it is also possible. Once you have found the Access Point of your partner service, UDDI's job is finished. It doesn't matter whether your partner has implemented WSDL or something else at their Access Point. There is another candidate already present called Web Services Conversations Language (WSCL) that defines Web Services in terms of conversations. WSDL and WSCL are not competing technologies, however detailed discussion of Web Services and conversations is beyond the scope of this article. Interested readers may begin searching further information about WSDL and WSCL from the UDDI.org web site.

The third possibility is a logical extension of the second. If your partner belongs to a particular community, they might have implemented a common business protocol on their Web Service that is generally understood in that community. Think of this common protocol as a set of business rules and interfaces (or a business language) that a particular business community has agreed upon. For example, there may be a common protocol for the hotel industry to expose room reservation and tariff related common functionality.

UDDI designers thought (and many companies that are investing into this technology are advertising) that one of the biggest value additions in B2B e-commerce will come when businesses start defining protocols according to the needs of particular industries. Readers who remember the development of the electronic data exchange industry will know that in the first instance, proprietary formats emerged which lead to the popularity of XML as open and flexible format for data exchange, and eventually uncountable XML based mark up languages were defined for almost every conceivable business sector ranging from health care to financial management.

This same process is expected to take place with UDDI as well. Business communities will define common protocols for interoperability. UDDI calls these protocols Fingerprints. Web Services are free to implement as many fingerprints as they feel like. For example our tourist site may implement fingerprints for the hotel and travel industries. You can even search for specific fingerprints within a UDDI registry.

Remember, all fingerprints are implemented as tModels in a UDDI record. Therefore, the tModel of our hotel example will contain methods that expose room reservation and tariff related business functionality.

How to Search through a UDDI Registry

Next question is how to put this UDDI database to work. The programming interface for UDDI consists of two parts: an Inquiry (search) API and a Publishing API. In this article, while discussing Inquiry API's, we will only discuss how to put a UDDI registry to work. The UDDI specification neither imposes restrictions nor makes suggestions about internal design and implementation of UDDI registries. This means that any database that can respond to the Inquiry and Publishing APIs in a manner that UDDI suggests is a UDDI registry.

This is essentially a business and architectural explanation of the Inquiry API that UDDI provides. We have the following methods available for searching through a UDDI registry. Remember the usage model of a UDDI registry; an ordinary Internet user as well as a B2B business can both make use of a UDDI registry (instead of mentioning the technical names, we have rather concentrated on the services that each of these methods provide):

1. Find the businesses of interest. This forms the logical start of most search operations. An Internet surfer will provide search criteria (such as products, product categories, key words, etc.) to begin a search. The intention of including this method is to allow preliminary searching, where further more precise searches will be carried out in subsequent steps (drilldown). Therefore this method returns a list of summarized results. The technical name for each summarized record is a BusinessInfo structure, which is the abbreviated form of the Business Entity structure of our earlier diagram.
2. The next logical step is to find the categories of services that a particular business offers (such as hotel reservation services that a travel and tourism B2B site may offer). Therefore, UDDI provides a method for finding an abbreviated list of services that any particular business offers. Smart web crawlers will use this method to find a list of all services, once the user has selected the business of their choice.
3. Find all tModels of interest. The user will, in this case, provide search criteria (say, business categorization, which is a very important search criterion), in response to which this method will return a list of tModels fulfilling those criteria. Users can search for specific fingerprints using this method. UDDI, while providing a mechanism for discovery of Web Services, at the same time also provides a very rich and extensible framework for business categorization. The UDDI specification allows development of specialized search services for specific industries on top of general registries.
4. Find bindings of interest within a particular business service. When we invoke this method, the UDDI registry will respond by sending a list of all binding templates that this business service contains. Recall from earlier that binding template provides an access point into a Web Service. A binding template may refer to a WSDL interface or a fingerprint.
   
   A business can implement several binding templates. This effectively means that each business is free to implement several WSDL interfaces and fingerprints simultaneously. For example, our tourist B2B site may choose to implement a particular fingerprint from the hotel industry (in addition to other WSDL based interfaces), so that all hotels that implement that fingerprint will be treated as our partner hotels. In this scenario, the find bindings method can help us check whether any particular hotel implements the fingerprint we are interested in or not.
5. Get the detail of any particular binding template. Normally users will invoke get methods to retrieve details of abbreviated results returned from find methods. Therefore, we have get methods available to retrieve details of all above-mentioned structures.

Any search through a UDDI registry will be a combination of these basic search methods. You can think of devising search conversations by using different sequences of primitive search methods. Implementing search conversations is the task of the smart and user-friendly GUIs that we mentioned earlier. UDDI operators will compete on the performance of their web front ends, so that ordinary Internet users can specify complex UDDI search structures and at the same time explore Web Services and invoke fingerprints.

UDDI: a Pragmatic Analysis

Before we conclude this article, we should briefly discuss how we could practically deploy these ideas today. It is possible to build, deploy, and publish fingerprints as tModels. There are a few UDDI registries already available (in order to get a list, visit UDDI.org). In addition, Microsoft, IBM, and Ariba (founder members of the UDDI project) have implemented public UDDI registries where users can publish their Web Services and fingerprints. Visit UDDI.org to check how to register and search. You can create your hotel's Web Services or fingerprints of the hotel industry and register them. This is all possible right now, although when you visit the registries of today you will not find many companies listed and several among the lists having incomplete entries. As e-commerce grows due to the flexibility and ease of Web Services, however, it will increasingly becoming difficult for companies to work outside the scope of Web Services and UDDI.

What is the sequence of events that will enable industries to put the idea of Web Services and UDDI to use? Is today's technology mature enough to allow us to participate in this growing phenomenon? In fact, organizations can create their own UDDI based publishing and search services right now. For this they need UDDI servers and clients. UDDI servers have started emerging and there is even an open source project that addresses this issue. The UDDI.org web site also provides more on this front. This, together with the fact that you can also create your own fingerprints, allows you to design UDDI based marketplaces. Imagine a UDDI registry dedicated to a particular industry (may be a textile or a metal exchange, or a Supply Chain Management solution selling desktop and laptop PC's, or a B2B application selling healthcare and related insurance plans). All you and your partners have to implement are the relevant fingerprints and UDDI search and publishing API's and your marketplace is functional (your UDDI server should handle all API functionality, so you and your partners are only left with the implementation of relevant fingerprints). Fortunately all the required building blocks are already available. All you need to get going is your business model. Visit Salcentral.com and XMethods.net in order to see what's already on the move.

Summary

We have discussed the operation of smart web crawlers and advantages of UDDI registries for both ordinary Internet users and owners of B2B portals. We also looked at the internal architecture of a UDDI registry, and some of the methods included in the UDDI search API. We concluded by recognizing the great importance of user-friendly front ends for UDDI operators, if they want to offer maximum advantage of Web Services to their site audience.

The extended PDF version of this article is available now.

Using SOAP as a UDDI Search Engine by Bilal Siddiqui
In this paper we will take a detailed look at the Description, Discovery and Integration process. For this purpose, we will discuss UDDI's usage models and its internal architecture. We will then see what's already present on existing UDDI registries and conduct a detailed analysis of SOAPClient.com, one of the leading UDDI related services of today. We will then move on with an in-depth discussion of what else UDDI offers as a search service and show that in fact it offers a lot that is yet to be brought to common Internet users. We present a conversation between an Internet user and a UDDI registry. The user specifies what service they want, and the registry responds. While doing this, we will design a Graphical User Interface (GUI), which will implement the User-UDDI conversation. We round off with a look at a smaller version of the GUI, for use on J2ME enabled devices, and a discussion of the technological and architectural issues involved in such a design process.

Purchase the extended PDF version of this article from:
our download site
or
Amazon.com

Adobe Acrobat format (PDF) - 315K
27 pages
Price: $10

Bilal Siddiqui is the co-founder of WaxSys, a company focused on simplifying E-Business. In his spare time, Bilal is a technology evangelist and plays with latest tools and know-how. He is also a frequently published author in tech magazines.

Printer-friendly version

Keep up to date with all the new articles and features on Web Services Architect:
Register for e-mail updates