The application layer
The application layer Clients, servers, peers
The TCP/IP application layer contains protocols that enable
applications to communicate. Computers connected to the Internet are end-systems or hosts
Olof Hagsand, NADA/KTH
olofh@nada.kth.se The TCP/IP application layer roughly maps to three OSI layers: (they “host” application programs running on them). Hosts are
traditionally divided into clients and servers - the difference
• Session: session establishment, dialog control, synchronization
January 20, 2005 nowadays unclear.
• Presentation: syntax and semantics of data: higher level data
But from a program point of view, it is easier:
structures
• Client program - requests a service.
• Application: application-specific information and protocols
• Server program - provides a service.
From its UNIX implementation roots, the definition of the the
application layer is: everything that is implemented in user space! • Peer - bot a client and a server program.
(not in the UNIX kernel).
1 2
0-0
The Socket Interface
Protocol message formats
The socket interface is used for programming applications with a
network component. When you transfer information from one host to another, they need
Approach 1: Binary fixed fields
to to understand each other’s data. (Presentation layer)
Sometimes called BSD sockets - it was first implemented in C in
BSD. Protocol messages are designed in different ways, some issues: Most common in the underlying layers of the TCP/IP stack.
Variants exist for most programming languages. • Performance - compact data for faster transmittal, easy to Examples: DNS, RIP, OSPFv2, BGP, RTP
Winsock is almost the same but not quite! parse by a computer. Predefines exactly what information is to be where in the message.
Other programming interfaces include: • Readability - Easy to read by humans: debugging, surveillance, The semantics is hard-coded into the application.
editing.
• Streams And its binary
• Common character sets - different languages, coding.
• Remote Procedure Calls (RPC)
• Alignment and byte ordering - Different CPU characteristics.
The sockets API is a de facto standard for network programming.
3 4 5
Pros & Cons
Example: DNS When you feel the urge to design a [...] complex binary
Binary fixed fields (cont)
application protocol, it is generally wise to lie down until
The DNS header, taken from RFC 1035.
the feeling passes
Requires common alignment (ie on 16, 32 or 64 -bit boundaries) 1 1 1 1 1 1
Eric Raymond: ‘‘The Art of UNIX Programming’’
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5
Requires byte-swapping: How the CPU loads its registers from +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
memory. | ID |
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+ Pros:
Two variants: |QR| Opcode |AA|TC|RD|RA| Z | RCODE |
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+ • Compact: Efficient computer processing
• Little endian (eg Intel): LSB (Least Significant Byte) first | QDCOUNT |
• Fixed syntax and simple semantics
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
• Big endian (eg Motorola): MSB (Most Significant Byte) first | ANCOUNT |
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+ Cons:
| NSCOUNT |
Network byte order is big endian → You need to byte-swap on i386 • Not extendable
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
PCs. | ARCOUNT |
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
• Not human readable.
• Byte order, alignment problems
6 7 8
TLV - (Type, Length, Value) Example: DHCP
A vendor extension field taken from RFC 2132.
Binary format usually used as an extensible part of a protocol.
Approach 2: Tree based
• Type: contains a predefined code, indicating what kind of data 3.5. Router Option
Data structured hierarchical - recursive structure. the value field contains. The router option specifies a list of IP addresses for routers on the
Both binary and textual variants. client’s subnet. Routers SHOULD be listed in order of preference.
• Length: Contains the size (in bytes) of the value field.
More or less formal specification defining the data-types E.g. XML The code for the router option is 3. The minimum length for the
• Value: Contains the payload. router option is 4 octets, and the length MUST always be a multiple
DTD. of 4.
Examples: IS-IS and OSPFv3, DHCP, and IP options.
Examples: TLV, ASN.1, XML. Code Len Address 1 Address 2
TLVs can be recursive (value field contains new TLVs). +-----+-----+-----+-----+-----+-----+-----+-----+--
| 3 | n | a1 | a2 | a3 | a4 | a1 | a2 | ...
But there is no notion of specification - must be added externally. +-----+-----+-----+-----+-----+-----+-----+-----+--
9 10 11
A tiny part of an SNMP definition: XML
Abstract Syntax Notation # 1 PDU ::=
SEQUENCE { Plain-text markup language: simple syntax, easy to parse.
request-id Integer32,
A general way to define data types. error-status INTEGER ( Definition declared externally by XML Schema or DTD.
noError(0),
ASN.1 is as powerful as a typed programming language. tooBig(1), Well suited for complex data formats with recursive and nested
In ASN.1 the type information is inherent in the data - no external noSuchName(2), structures.
badValue(3),
specification necessary. readOnly(4), Cons mainly its textual nature: parsing can be inefficient.
Used frequently in ISO protocols, but also to a certain extent in ...
TCP/IP protocols.
inconsistentName(18)
), Eva
Some examples: SNMP, UMTS, LDAP, NFSv4 and many security Phil
error-index INTEGER (0..max-bindings),
protocols. variable-bindings VarBindList Reminder
} Remember to go to the store!
12 13 14
RFC 822 based text protocols (con’t)
RFC 822 based text protocols (con’t)
For example:
Approach 3: RFC 822 formats name = elements crlf a rule Another example; In RFC 2048, the HTTP URL is defined as:
Classical Internet format described by BNF (Backus-Naur Form) - crlf = %d13.d10 characters to end a line http URL = ‘‘http:’’ ‘‘//’’ host [ ‘‘:’’ port ] [abs path
derived from context-free grammars. ‘‘literal’’ a string, case insensitive host = A legal Internet host domain name or IP
Several RFCs describes the actual syntax description: RFC 822, element1 / element2 an alternative address (in dotted-decimal form) as defined by
RFC 2068, RFC 2234, now called ABNF - Augmented BNF. (element1 element2) a strict sequence Section 2.1 of RFC 1123
RFC 822 is syntax-heavy: keywords are introduced for parsing, DIGIT = %x30-39 a range of characters port = *DIGIT
requires specific parsers. *element element repetition abs path = ‘‘/’’ rel path
[foo bar] optional elements rel path = [ path ] [ ‘‘;’’ params ] [ ‘‘?’’ query ]
. . . and more . . .
15 16 17
TELNET - Terminal Network
Specific applications/protocols
(TCP port 23, text) Virtual Terminal local terminal appears to be
• telnet a terminal on a remote system
Pros & Cons
• http It is a nice tool to test other text-based protocols (HTTP, SMTP,
• Pros: FTP, etc)
• tftp
– Easy to extend and flexible.
• ftp Good example of interactive application
– Human readable (easy to debug)
• smtp • Tinygrams leading to silly window syndrome:
• Cons:
• snmp • Nagle’s algorithm
– Not compact.
• rtp • Delayed ack, etc
– Syntax-heavy: may require complex parsers.
• sip Control: simple options (control bytes have first bit set)
TELNET is security challenged: use TELNET with Kerberos or
• Others: Instant Messaging, Peer-to-peer, Distributed gaming.
SSH!
18 19 20
HTTP - example
-> GET /stuff/blah.html HTTP/1.1
-> Host: zipf.pilsnet.sunet.se
HTTP ->
->
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.5) Gecko/20031214 Firebird/0.7
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,video/x-mng,image/png,image/jpeg,image/gif;q=0.2,*/*;q=0.1
-> Accept-Language: en-us,en;q=0.5
(TCP port 80, RFC 2616, ABNF data)
->
->
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Some HTTP commands
-> Keep-Alive: 300
The Hypertext Transfer Protocol is the main protocol used to -> Connection: keep-alive
• GET http url: Download an http resource.
21 22 23
HTTP 1/1 persistent connections TFTP - Trivial File Transfer Protocol
(Text-based, UDP port 69, RFC 1350)
In HTTP 1/0, all HTTP requests generated a new TCP connection.
Very simple protocol to transfer files.
But most html documents contain sub-parts → one TCP
Some HTTP status codes (con’t)
Character coding: netascii(like telnet) or binary.
connection for each sub-request.
Some examples: But TCP congestion control is made for longer connections → they Stop-and-go protocol: send datagram, wait for ack.
can adapt to congestion in the network. Small implementations: typically on boot PROMS for small devices
• 200 Ok
and diskless clients.
• 404 Not found When http traffic grew when the web exploded, these small flows
were said to kill the Internet! Five message types:
• 301 Moved Permanently • RRQ - Read ReQuest
HTTP 1/1 supports persistent connections: keep the TCP
• 500 Internal Server Error connection during the complete session: send all requests on the • WRQ - Write ReQuest
same TCP connection. • DATA
Now, these longer TCP connection can perform congestion control • ACK
algorithm in a proper way.
• ERROR
24 25 26
FTP - File Transfer Protocol Some FTP commands
FTP modes
(Text-based, TCP ports 20 and 21, RFC 959) Examples of FTP control commands (sent on control channel):
FTP can run in two modes- active mode and passive mode. This
FTP is a more elaborate file transfer protocol. refers to whether the ftp server will start the data connection or • CWD Change working directory
FTP conducts its sessions in clear text. not. • RMD Remove directory
FTP uses two TCP connections: • active: The server will start the TCP session for the data • PWD Print working directory
connection, thereby connecting to the client to a port and IP
• The control connection - exchange commands and their replies. • TYPE [I|A|E|L ] Set the data transfer type
specified by the client. (May not work if client is behind NAT)
TCP session initiated by the client to the server on port 21.
• passive: The server will not start a TCP session. Instead, the • RETR Download a file.
• The data connection - to transfer data in a specified mode and
client will create a TCP session to the server, to a port and IP • STOR Upload a file.
type. Data transferred may be a part of a file, an entire file or
specified by the server.
a number of files. • LIST Download the current working directory’s content list.
27 28 29
Some FTP status codes SMTP - Simple Mail Transfer Protocol
As in HTTP, FTP has a variety of status codes: (Text-based, TCP port 25, RFC 2821) SMTP (cont)
• 1xx Positive Preliminary reply SMTP the protocol to transfer email from hosts to mail servers and
The requested action is being initiated; expect another reply before proceeding Electronic mail is different from the previous protocols in its
with a new command. between mail servers.
delayed delivery in several steps:
• 2xx Positive Completion reply Terminology:
The requested action has been successfully completed. A new request may be • Spooling from sending host to first MTA.
initiated.
• User Agent(UA) - end-hosts.
• 3xx Positive Intermediate reply • Relaying by intermediate MTAs.
The command has been accepted but the requested action is waiting for further • Mail Transfer Agent (MTA) - mail servers.
information before being completed. • Downloading of email by receiving host using other protocols:
• 4xx Transient Negative Completion reply POPv3 (Post Office Protocol) or IMAPv3 (Interactive Mail
The command was not accepted and the requested action did not take place,
but the error condition is temporary and the action may be requested again.
Addressing: @ Access Protocol)
• 5xx Permanent Negative Completion reply
This results in a a DNS MX request for , giving a
The command was not accepted and the requested action did not take place. name of the MTA to transfer the message to.
30 31 32
MIME - Multipurpose Internet Mail Extensions
SMTP syntax
Classical email messages must be written in US-ASCII (7-bit). So how does it work?
Like HTTP and FTP, SMTP has special commands and status
codes. What does this imply?
Related header fields:
• HELO : MIME aims at redefining the format of messages to allow for:
• Content-Type - what kind of data the content carries.
• MAIL FROM : Sender email address • textual message bodies in character sets other than US-ASCII, Some examples: text/plain, text/html, audio, video,
• an extensible set of different formats for non-textual message application/pdf, extension-token, and multipart.
• RCPT TO : Recipient email address
bodies, • Content-Transfer-Encoding - how data is encoded.
• DATA: Tells the email server that data follows.
• multi-part message bodies, and Some examples: 7bit, 8bit, binary, quoted-printable,
• QUIT: Immediately close the connection. base64,...
• textual header information in character sets other than
The status codes are similar to that of HTTP and FTP. US-ASCII
33 34 35
SNMP Simple Network Management Protocol
(TCP, ASN.1)
• It is complex to build internetworks and we need to manage
them. Real-time multimedia Signaling
– Monitoring Time-sensitive, interactive applications: (eg, telephony). So, RTP can be used to transfer time-sensitive data streams.
– Debugging Use RTP- Real-Time Protocol.
But what about signaling: how to set up sessions:
– Control routers and other network devices Limited time-sensitivity: Streaming protocols.
Use RSTP (Real-Time Streaming Protocol) • SIP - Session Initization Protocol
• SNMP - Internet management
Non-time sensitive: Transfer the data using file transfer. • H.323
– No special control messages use TCP/IP itself
– Management is on TCP/IP application level
– Same protocol is used for all managed devices
– If IP does not work correctly,...
36 37 38
SIP Example
From RFC 3261:
SIP softphone
|
proxy
|
proxy
|
SIP Phone
|
SIP message example
(TCP or UDP port 5060, ABNF) | INVITE F1 | | |
INVITE sip:000730631661@kth.se SIP/2.0
|--------------->| INVITE F2 | |
Via: SIP/2.0/UDP 192.36.125.167:5060;branch=z9hG4bK0e4415ea
Terminology is similar to SMTP, but is a synchronous protocol (no | 100 Trying F3 |--------------->| INVITE F4 |
From: "6534" ;tag=000e38a3b7e8001d597d1d53-1bfa7620
||
delays). | |
| | 180 Ringing F7 | User-Agent: CSCO/6
| 200 OK F11 | |
| ACK F12 | Expires: 180
SIP uses transactions, usually three-way (as TCP connections). |------------------------------------------------->| Content-Type: application/sdp
| Media Session | Content-Length: 251
Example: Accept: application/sdp
||
INVITE → 200 OK → ACK | BYE F13 |
||
39 40 41
IM - Instant Messaging
Peer-to-peer file-sharing applications
On-line messaging and presence information using a central server Peer-to-peer file-sharing applications (cont)
and many connected clients. Example of content-distribition (file-sharing) using peer-to-peer
techniques. Some are completely decentralized (GnuTella), encrypts data
Some systems: AOL IM/ICQ, MSN Messenger, Yahoo Messenger
(FreeNet).
An IM system typically has the following features: Build overlays – virtual networks on top of physical network.
Overlay links are TCP/UDP connections. BitTorrent, for example, works closely with HTTP - splitting up an
• Buddy list HTTP transfer in slices, distributing the download from one
Usually, actual data transfer is direct between hosts (peer-to-peer),
• Chat, Images, Sounds, File-sharing originator to many clients working in unison.
often using HTTP.
Many rely on distributed hash lookup functions to make fast
• Real-time talk and video Some have central registry (index of where files are) (Napster).
queries and lookup of data.
Most protocols are proprietary. But SIP has messaging extensions Others (eg KaZaa) have distributed registry: some nodes with good
Some of the routing problems are similar to real (physical) routing,
(SIMPLE). network connections, no NAT, and large resources turn into
but on a higher level.
A special feature is to serve many small messages in a short time, supernodes. All clients connect to a supernode.
and to manage presence information.
42 43 44
Skype Distributed games
Detour: NAT traversal (Encrypted, TCP/UDP) Some of the best-known distributed games are interactive and
Skype is a VoIP tool using peer-to-peer techniques for name-lookup. real-time: Doom, Quake, Counter-Strike, Half-life, etc.
Nowadays, most hosts are behind NAT (Network Address Some issues are:
Skype is a completely closed system - no open interfaces, not even
Translation) boxes
which RFCs are implemented → No interoperation possible. • Low latency: “low pingers” win fights. Usually small UDP
NATs translate global IP addresses to local, and extends the You could say this violates the Internet spirit. packets.
address space using TCP/UDP ports.
• Uses high compression: iLBC coding (≈ 10x compression of • Textures and geometric information preloaded: only deltas
One peer behind NAT: possible to initiate connection from behind audio data) distributed.
a NAT.
• Name lookup using same infrastructure as KaZaa: nodes and • Movement of 3D graphics may use “dead reckoning”: no need
Both peers behind NATs: difficult to communicate directly. supernodes. to send updates on all geometric movements: use motion
Solution: For UDP, exploit some regularities of NATs (reuse of equations instead.
• NAT traversal techniques using UDP, TCP or “bounce”
same ports, etc). Or use a non-NAT peer as “protocol bouncer”. connections via supernodes. • All communication via central server, synchronizes and resolves
• End-to-end RSA encryption events (who wins a fight).
45 46 47