Chapter 7: Network Communication...................................................................................................
URLS..................................................................................................................................................................................
WORD SEARCHING A URL........................................................................................................................................
DEALING WITH CONTENT OTHER THAN SIMPLE TEXT...............................................................................
A LINK CHECKING APPLET......................................................................................................................................
USING SOCKETS............................................................................................................................................................
Socket Basics...............................................................................................................................................................
The Snitch Applet.......................................................................................................................................................
InetAddress..................................................................................................................................................................
TO BLOCK, OR NOT TO BLOCK...............................................................................................................................
NETWORK IO IN THE AGENTSERVER..................................................................................................................
The Message Classes.................................................................................................................................................
CONCLUSION...............................................................................................................................................................
EXERCISES...................................................................................................................................................................
Chapter 7: Network Communication
Every programming tool has a dirty little secret that makes using it less pleasant than the walk in the park the marketing brochures promised. Basic has its spaghetti code, and C its wild pointers. Java is no different. The dirty little secret of Java applet writing is that browser security mechanisms make it nearly impossible to write a useful Java applet that doesn’t connect back via sockets to a daemon on the server. Specifically, browsers can, and do, prevent applets from doing any file IO. This means that any persistence that an applet requires has to be implemented in a server application (not an applet) that the applet talks to over the net.
Java provides network communication through the package java.net. This package contains a number of useful classes. The basic ones that we’re going to use and talk about are URL, Socket, ServerSocket and InetAddress
A URL, or Uniform? (Universal?) Resource Locator, is basically a network location. It tells you not only where something is, but what it is. As an example, consider my home page URL:
http://www.channel1.com/users/ajrodley/index.html
Briefly, this says that to use this ‘document’, you connect to the http server on the machine named www.channel1.com, and tell it to send us a stream of data made out of the file “/users/ajrodley/index.html”.
Java URLs take this concept of a net location one step beyond. After all, a URL by itself can be easily represented by a String. A Java URL encompasses not just the address, but the object at that address.
Consider the example of Figure 7-1, which scans a document at a URL for a particular word.
Listing 7-1: an applet that scans a document for a word.
<ch7_fig1.java>
Listing 7-1a: The HTML page that loads the word finder applet.
<ch7_fig1.html>
The HTML page provides both the URL of a document to scan and the word we wish to find. Notice a couple of things. First, we cheat by putting the whole body of our applet in the start method. This is bad practice. For applets that do any amount of processing, the body of the applet should run in a separate thread. Second, the document we’re scanning isn’t HTML. We’ll talk about that later.
Most of the functionality of this applet is embodied in the call to URL.getContent. When the file we’re pointing at is simple text, what we get back from the call to getContent is a simple string that holds the entire text of that file.
Having read our whole file into a String, there are now two ways to find our word within that string. The first way is an exact match via String.indexOf. This searches the whole string, whitespace included, for the word.
The second, more useful way uses the StringTokenizer to turn that string into a Vector of lines. Then it turns each line into a Vector of words and compares each member of that Vector to the desired word. The two methods, indexOf and compareTo will often find different things. indexOf will find the word within another word, while compareTo won’t. compareTo combined with toUpperCase matches the words regardless of case.
The final thing to notice is that we haven’t talked at all about sockets, protocols or daemons. All of the grunt work of connecting over the network and retrieving the file is handled entirely within URL.getContent.
Dealing With Content Other Than Simple Text
If you point the applet of figure 7-1 at a file with .htm, or .html extension, something very disturbing happens. getContent throws a ClassNotFoundException. This is because there is no content handler for content of type HTML. The steps URL.getContent goes through to deliver our String in figure 1 are:
make a connection over the net via sockets
create a stream of characters flowing from the server to the applet
figure out that the stream is simple text and turn that stream of characters into a String object
The problem with using this applet on HTML content is that Java figures out that the stream is HTML (not simple text) but it doesn’t know how to create a sensible object from a stream of HTML text. This is unfortunate, but not insurmountable. Of the three mechanisms mentioned above, the first two are still available to us, regardless of the type of the URL’s content. We just have to turn deal with the stream ourselves, rather than having getContent turn it into a String.
To do the same search and have it work whatever the content type, we have to modify the applet from figure 1 as shown in figure 2:
Listing 7-2: Search HTML documents for a word.
<ch7_fig2.java>
We get rid of the call to getContent, and rework the FindWord method to suck the HTML text over the network one line at a time. The InputStream that getContent uses to create an object appropriate to the URL’s content is available to us via URL.openStream. Then we turn that InputStream into a DataInputStream so that we can read line by line rather than InputStream’s byte or byte array read. From there, we use the same StringTokenizer techniques as in Figure 7-1 to match the actual word.
From these two applets, we can develop the rudimentary link-checking applet of Figure 7-3.
Listing 7-3: A link-checking applet.
<ch7_fig3.java>
Figure 7-1: Link checker running against my home page.
<ch7_fig1.pcx >
This applet has a couple of weaknesses. The first is in parsing the HTML. We use the same basic technique, starting with three strings that indicate “links”:
“<A HREF=“
“<IMG SRC=“
“<applet code=“
This works pretty well, but it’s a far cry from the sort of rigorous syntax checking a commercial product would need to do.
The second big weakness in this is a by-product of the structure of the web itself - circular references. In the most basic case, if you have two pages that contain links to one another, an unsophisticated web crawler will sit spinning in an endless loop. Our checker takes a number of steps to try to prevent this.
Sockets are at the heart of almost every instance of Internet communication. When, for instance, you point your Web browser at http://java.sun.com, the browser uses the socket interface to connect to a port on java.sun.com. Sockets are a simple, old, tried-and-true, technology that make network programming very easy.
There are two ends to each socket conversation: server and client. The server end is embodied in the ServerSocket class, while the client end is embodied in Socket. These two ends go through a specific set of steps to setup, conduct and terminate a conversation, as follows:
Table 7-1: Steps in a client-server conversation.
Server Client
Instantiate ServerSocket passing a local port number. This creates the socket and binds it to that local port number.
Instantiate Socket passing a server name and port number. This creates a client end socket and connects it to the named port, on the named host. When this call returns, the two ends are connected.
The ServerSocket constructor returns a Socket, that can now be used for IO. Most applications will spawn a new thread to read and write this Socket. The ServerSocket can continue to ‘accept’ connections on the original socket.
From this point on, both server app and client applet can read and write the connected sockets.
How does one side know when the conversation is over? Many protocols call for there to be a ‘goodbye’ but depending on something like that won’t get you very far. Lost connections are a fact of life. Fortunately, Java throws an IOException in almost any case where the network connection has been interrupted. You must catch and handle IOExceptions properly to write useable network communications code.
The Snitch Applet
With those basics well in hand, let’s construct an application/applet combo that does some very simple socket communication. The purpose of this applet/application combination is to record the date/time, URL and ip address of the user whenever someone hits the HTML page this applet is embedded in. This is one of the holy grails of web publishers, to be able to know who is hitting their page and when. This applet is unusual in that it has NO user interface. The user never sees it.
The theory behind the system is simple enough. The server Java application Snitch, is running all the time on the server accepting connections on port 1038. When a user loads the page with our applet in it, the applet starts up, gets the page URL, the host name and ip address, and packages all that information in a message. Then it connects to the server socket and send the message to the server, which stores it in a file from where it can be retrieved and analyzed. Figure 7-4 shows the snitcher applet.
Listing 7-4: The snitcher applet.
<ch7_fig4.java>
InetAddress is the interface between Java and the network name service. You can use it to turn a host name into an IP address or vice versa. The Snitcher applet uses the static method InetAddress.getLocalHost to get a complete description of the machine that the applet is running on. Notice that all we need do to get the hostname/ip address is call InetAddress.toString. This is a recurring theme in Java. It is also what happens if you append a non-String object to a String via the ‘+’ operator as in:
InetAddress in = InetAddress.getLocalHost();
String s = “blah blah blah”+in;
Java calls in.toString in order to append it to the first string.
Snitcher also uses getDocumentBase to find the machine on which the Snitch server application is running. This is not an arbitrary choice, but a requirement of some browser SecurityManager implementations - applets can only open socket connections to the server from which the HTML page was loaded.
Listing 7-5: The Snitch application.
<Snitch.java >
A close look at Snitch.java reveals a basic skeleton that all server applications follow. At the top level is a thread that does almost nothing except create a Frame window for accepting user input, and another thread to accept connections to a ServerSocket. Each time a client applet connects to the server socket, the acceptor thread creates a new thread to read the port and deal with whatever the client sends us, in this case, writing a line of text to the log file.
Like the File class, the Socket class provides two methods, getInputStream and getOutputStream, that provide a base object through which we can do whatever style of IO we wish. The AgentServer, for the most part, does non-delimited byte-level IO using the bare InputStream. Snitch, on the other hand, receives CRLF delimited lines of text from its client applets. Thus, it needs to create a DataInputStream from the Socket’s bare InputStream, and use DataInputStream.readLine rather than InputStream.read. You will find that almost all IO operates this way. You take a bare InputStream or OutputStream, then create a more sophisticated, higher level stream, like DataInputStream, using that bare stream.
The other big difference, between Snitch and more complicated server apps like AgentServer is that we do only a single readLine, then close the socket, where most servers keep reading the Socket until the client applet disconnects.
One of the key characteristics of any IO operation is whether or not it blocks. If you call InputStream.readLine, no matter how many bytes it does read, it will not return until it reads a line terminator. It ‘blocks’ until the line terminator is read. If readLine were non-blocking, it would return immediately whether or not it had read a line terminator.
Many coders, especially those who grew up in the bad old days of single threading, prefer to write their communication code as non-blocking. In single threaded systems there are very good reasons for this, one being that code that blocks, often fails to unblock.
That reasoning doesn’t hold up in Java. Any thread which has blocked should be able to be unblocked by calling Thread.stop (throwing a ThreadDeath at it). Java also doesn’t support many of the system calls, select and available to name two, that UNIX coders used to rely on to write non-blocking io.
Almost all Java IO calls, socket connect, stream read and stream write, block. Some allow the operation to timeout, but for the most part, you are literally required to thread and block.
We now have everything we need to write the network IO portion of the agent system. Listing 7-6 shows the final AgentServer code.
Listing 7-6: The final AgentServer.
<AgentServer.java>
The network communications of the AgentServer follow the same skeletal form as the example of figure 7-? The top level thread creates a Frame window for accepting user input, and another thread to accept connections to a ServerSocket. Each time a client connects to the server socket, the acceptor thread creates a new thread to read the port and deal with whatever the client sends us. The main difference here is that the AgentServer has a huge set of message handling code (described below) that the Snitch doesn’t need.
The AgentLauncher’s network IO is mostly confined to a single private class, AgentDispatcher. AgentLaunchers can only send two message types, dispatch and kill, each of which gets its own method. They can also only receive two message types, start and result. Each of these causes a call back into the AgentLauncher once the message has been parsed.
Listing 7-7: The AgentDispatcher class.
<ch7_fig7.java>
There are two different types of connections in the Agent system: the single message connections between AgentServers, and the persistent connection between the AgentLauncher and the AgentServer. We want to try to avoid persistent connections when possible because a quiet connection is a waste of resources for one thing, but more importantly, a uselessly lingering connection is an invitation to the kind of network interruption that will end up throwing an IOException.
We use a persistent connection to the AgentLauncher for two reasons. First, we can use a broken connection as a sign that the client has gone away and we can ‘kill’ any of his Agents left out in the field.
Second, it’s possible to restrict an applets ability to create a server socket while still allowing it to open client sockets. In order to use single-messaging back to the AgentLauncher from the dispatching AgentServer, the AgentLauncher would have to create, and accept on, a ServerSocket but with ServerSocket creation limited by the SecurityManager, this wouldn’t work.
With the socket handling structure we’ve already outlined, we now have the parts of the agent network connected. What we need to do is to define a protocol and a set of messages that will get all these connected computers working together.
Any messaging system can be split into two pieces: the protocol and the actual message structure. Table 7-2 shows the messages supported by the AgentSystem.
Table 7-2: The supported messages.
QueryAgentList AgentLauncher -> dispatching AgentServer Send me the list of agents that this AgentServer can dispatch
AgentList dispatching AgentServer -> AgentLauncher Here is the list of agents that this AgentServer can dispatch
Dispatch AgentLauncher->dispatching AgentServer Dispatch the named class to all your servers
Load AgentServer->AgentServer Load and run the supplied class
Kill AgentLauncher->AgentServer Kill the named agent
Start AgentServer->dispatching AgentServer -> AgentLauncher The named agent has started work
Result AgentServer->dispatching AgentServer -> AgentLauncher The named agent is reporting results in the named URL.
Table 7-2 also gives a rough idea of the protocol the system uses. The QueryAgentList and AgentList messages are the only query-response pairs in the system. All the other messages can arrive in any order. In the normal course of events, the AgentLauncher and its dispatching AgentServer exchange a QueryAgentList/AgentList message pair. Then, the AgentLauncher sends its AgentServer a Dispatch message. At that point, the dispatching AgentServer sends out a number of Load messages. As servers process the Load messages, and start running the Agents, they start sending Start messages back to the AgentLauncher. As each Agent finishes, it sends back a result message.
The other half of any messaging system is the actual message structure. Each message has a header that tells what type of message it is, and how long it is. The header is followed by a series of fields. Each field has a header that tells what it is, and how long the data portion of the field is. The order of the fields IS significant.
Table 7-3: The message structure.
Message type 4 bytes ascii message type description, i.e. “Disp”
Size of length 4 bytes ascii integer describing the length of the length field
length Size of length bytes ascii integer describing the size of the entire message
Field type 4 bytes ascii field type description, i.e. “Clas”
Size of length 4 bytes ascii integer describing the length of the length field
length Size of length bytes ascii integer describing the length of the data portion of the field
Data length bytes the actual data of the message, i.e. the bytecodes transmitted when sending a load message, or the string URL when reporting results.
It only makes sense for the AgentServer and AgentLauncher to share the message classes, so we split them off in their own package, agent.util. Listing 7-8 shows the actual implementation of the message classes.
Listing 7-8: The message classes.
<Message.java>
<QueryAgentListMessage.java>
<AgentListMessage.java>
<DispatchMessage.java>
<LoadMessage.java>
<ResultMessage.java>
<KillMessage.java>
<StartMessage.java>
We create one public class for each message. This class contains methods for both the message construction and parsing. To create a message from scratch, we use the constructor which takes the constituent fields as arguments, then call getMessageBytes to get a lump of bytes suitable for network transmission. To parse a message, we call the do-nothing constructor, then call parse to break the message into constituent fields.
The message classes all subclass the base abstract class Message. Message, for the most part, merely enforces an interface, in this case, the getMessageBytes method for getting a lump of bytes to transmit over the network. If not for the utility methods, makePrefix and zeroPadToLength, Message could have been written as an interface rather than as a class.
Most of the message bodies are fairly monotonous string and byte-array manipulations. When sending a message, the theory of message creation is that:
the constructor stores the distinct fields
createMessage fills the byte array msg with the actual message text
getMessageBytes returns the msg byte array which can be sent over the net
On the parsing end, the steps are even simpler:
the constructor does nothing
parse takes the message as a byte array and reduces it to the constituent fields
the constituent fields, as public members, can now be used by whoever called Message.parse
Writing the messages this way means that the receiver of a message only needs to read the first four bytes, instantiate the proper message class based on those four bytes, then call parse to break the message up properly.
As written, the message classes are not particularly efficient, as they stage message data a couple of times before getting it into, or out of, the actual msg byte array. This is a debugging tool that allows us to dump constituent parts at various stages of construction/parsing.
There are a few things to note about the data carried by the messages. There is one ID that uniquely labels a ‘run’ of a particular agent. This ID is generated on the AgentLauncher, and, for this version, consists of the lead class file name, and a date/time stamp. This should be sufficient to ensure uniqueness.
In the load message, the ip address of the dispatching AgentServer appears. This follows the Agent all the way around the network, so that, although AgentServer C may get this load message from AgentServer B, he’ll connect directly back to dispatching AgentServer A to report results.
Java makes network communication easy, through a set of simple classes - URL, ServerSocket, Socket, and InetAddress - that abstract the important concepts in Internetworking. While URLs provide some high-level functionality, through getContent, you can easily program right down to the lowest levels using Sockets.
Using these basic tools, we’ve constructed a functional system of cooperating Java objects. While security restrictions force some inelegance in the design, our purpose has been achieved with a modular, portable and fairly readable implementation.
1. What is a URL?
2. URL and InetAddress both embody the idea of where things are on the net. What is the difference between them?
3. Why are network communications essential for writing useful applets?
4. Does the example of figure 7-1 use sockets?
5. Why is the LinkFollower class of figure 7-5 dangerous?
6. There are no methods within the Socket class for reading and writing the socket. What classes do you actually use to read and write against a socket?
7. In the example of figure 7-4, does the Skip button actually stop the search?
8. Rewrite the base Message class as an interface.
9. Rewrite the AgentServer’s Vector of running agents as a HashTable. How is it better than the original Vector?