Startup and Configuration

The proxy reads its configuration and starts listening on TCP sockets (address/port couples) specified by listen-on configuration section, see listen-on(5). If support of transparent connections (i.e., connections made directly to a HTTP server and redirected to the proxy by IP Filter as described in transparency(7)) is requested by item transparent in section listen-on, the corresponding NAT redirections are established during proxy startup and removed upon exit.

Format of the configuration file is described in http-proxy.cfg(5). General syntax of Kernun configuration files is explained in configuration(7). Program test-http tests syntax and partially semantics of configuration; for test expression syntax, see test-expr(5).

Access Control

Http-proxy uses three-phase ACLs, see access-control(7). The first phase, session-acl is checked once for each client connection. It permits or denies client access and sets some connection parameters. The second phase, request-acl is checked for each HTTP request after the request headers are received from the client, but before anything is sent to the server. It decides about permitting/denying the request and it can also set some request parameters. Note that there can be several requests per connection if persistent connections are used. The third phase, doc-acl is checked for each HTTP request after the response headers are received from the server, but before the response is sent to the client.

Connection Establishment

When a connection from an HTTP client arrives, the configuration is searched for a matching session-acl. If the ACL says that the connection should be denied or there is no matching ACL, the proxy does not communicate with the client and closes the connection immediately. In addition to the generic ACL conditions and actions described in access-control(7), some Http-proxy-specific conditions and parameters can be set.

If the proxy sends a response to the client, but the client is still sending some data, it may be necessary not to close the read side of the connection for some time (see RFC2616 sect. 10.4 for details). Configuration item linger-time sets the time which the proxy should wait before closing the read side of the client connection.

It is possible to set idle timeouts for request and response by item idle-timeout. If no data are received from the client or the server for more than idle-timeout seconds, the request fails.

Items client-keepalive and server-keepalive define whether persistent connections are to be used on the client and server sides of the proxy. It is possible to limit the number of requests per connection or to disable persistent connections entirely. Timeouts for closing an idle connection can be set too.

The language used by the proxy in error messages sent to the client is defined by item language.

It is possible to forward all requests to some other proxy instead of sending them directly to origin servers. The next-hop proxy is set by hand-off.

Item ssl switches on SSL/TLS on the client connection and sets various SSL/TLS parameters. If the connection from the client uses SSL/TLS, item client-cert-match defines the acceptable client certificates. If the client certificate does not pass the test, SSL/TLS connection establishment fails and the connection is closed.

When http-proxy is used as an authentication proxy for accessing protected HTTP servers, parameters of the client authentication are set by item aproxy.

Request Processing

For each request from the client, the proxy reads the request line and headers. Then it finds the appropriate request-acl. If the ACL says that the request should be denied, the user should authorize, or there is no matching ACL, the proxy sends back a reply informing the user that the request has been denied. In addition to the generic ACL conditions and actions described in access-control(7), many Http-proxy-specific conditions and parameters can be set.

Items request-uri, request-method, request-scheme, request-path, and request-version match values from the first line of the request. Note that the server hostname and port is matched by the standard ACL condition server. In a non-transparent proxy, the server hostname and port for matching both request-uri and server are taken from the request URI received from the client. If request-acl.host-hdr-transp is set, the Host header is used instead. This allows a request URI without a server name in a non-transparent request, which can occur if a request is transparently redirected differently than by the Kernun's own transparency. In a transparent proxy, the server hostname and port are taken from the Host request header. If Host is missing in the request (possible in HTTP/1.0) then the original destination IP address and port of the connection from the client is used instead. The server address and port used for matching is also the address where the proxy connects to. Especially, in a transparent proxy this may differ from the original destination address used by the client. Combination of a server and an initial part of path can be matched against a blacklist using blacklist item. The name of the blacklist database file is specified by blacklist-db. Utilities for working with blacklists are mkblacklist(1), printblacklist(1), and resolveblacklist(1). The set of categories assigned to the request URI can be matched by item clear-web-db.

Sometimes different settings are needed for some types of clients due to various errors and incompatibilities in web browsers. Therefore, the value of User-Agent HTTP header may be used for ACL matching (item user-agent).

If the client connection uses SSL/TLS, the values from the client certificate are compared to item client-cert-match during ACL matching. An ACL with client-cert-match is never used for plaintext HTTP connections.

If aproxy is configured in the session-acl, it is possible to use item aproxy-user in order to match the user and the group authenticated by AProxy. If aproxy is not configured, aproxy-user none matches.

It is possible to change the whole request URI (e.g., http://www.tns.cz/index.html). The URI is matched with a regular expression. A matching URI can be rewritten to some other string, as defined by item rewrite. The request processing continues with the current ACL, even if the new URI does not satisfy the conditions of this ACL, because request-acl matching is done only once and with the original URI. It is possible to specify a redirect permanent or redirect temporary in a rewrite. Then the proxy will not fetch the rewritten URI, but it will return a HTTP redirect response (status code 301 or 302) to the client.

Item plug-to changes the server address which the proxy connects to, but it does not change the content of Host header. It is possible to change Host header (and also the server in request URI in the case of hand-off) by http-host.

Items hand-off and language allow to overwrite the values from session-acl for a single request.

Items file-response and program-response generate a response locally by the proxy. Unlike the replace-response item in doc-acl described below, the proxy does not contact the origin server and generates the response immediately. See the section called “Program-Generated Responses” for details about the proxy-to-program interface.

Item select-optimization influences internal handling of network communication. The proxy repeatedly checks its client and server network connections for a possibility to read or write data. When a connection is ready, a piece of data is sent or received. It is more efficient to try several send or receive operations on the connections that have been ready recently before checking all existing connections again. The number of such retries is controlled by select-optimization. It may improve the proxy performance if set to a small positive value, for example 10.

Request and response headers may be filtered by items allow-req-hdr and allow-resp-hdr. These items define names of headers that will be passed by the proxy. All other headers will be deleted from the request or response. It is also possible to reject requests with request, status, and header lines not matching items req-line-check, req-hdr-check, status-line-check, and resp-hdr-check.

The proxy may add Via HTTP headers to requests and responses. These headers inform about proxies which the request or response passed through. It may be useful to track problems with a proxy, but sometimes the administrator wants to hide information that the proxy is present. Hence the use of Via headers may be configured by items request-via and response-via.

Item request-time places an upper limit on the request handling time. It eliminates stuck clients and servers and help against some DoS attacks.

Item auth-req causes that the proxy responds with 407 Proxy Authentication Required and sets the authentication realm sent to the client.

Maximum amount of data transferred between the client and the server may be limited by item max-bytes separately for each direction (client to server and server to client). Data filtration can change the size of data significantly, therefore the limits are set separately for client and server connections.

In some situations, e.g., when chunked transfer encoding is in use, the proxy buffers incoming data and sends them only when the buffer is full. There are applications which require all data to be forwarded immediately. For such situations, item flush switches buffering off.

Item ssl switches on SSL/TLS on the server connection and sets various SSL/TLS parameters. If the connection to the server uses SSL/TLS, item server-cert-match defines the acceptable server certificates. If the server certificate does not pass the test, SSL/TLS connection establishment fails and the request terminates with an error.

After request-acl is processed, the request is forwarded to a server. When the servers answers with a status line and response headers, the proxy finds the appropriate doc-acl. In addition to the generic ACL conditions and actions described in access-control(7), some Http-proxy-specific conditions and parameters can be set.

Items request-scheme, request-path, and blacklist have the same meaning as in request-acl. Item mime-type provides matching of response document content type. The proxy provides three methods of detecting the content type: content-type (from the Content-Type response header), extension (matching request URI suffix with information from mime-types), and magic (guessing the type from an initial part of response data, using the same algorithm as in the standard utility file). Selection and priorities of the methods are defined by http-proxy.doctype-identification, session-acl.doctype-ident-order, and request-acl.doctype-ident-order. The first successful method defines the type. If no method succeeds, the type will be represented by an empty string. Maximum size of data scanned by magic method can be changed by http-proxy.doctype-identification.magic.

The content type, i.e., the Content-Type header sent to the client, can be forced by set-mime-type, or set to the content type discovered for mime-type matching by force-doctype-ident. Otherwise, the header is left unchanged.

It is possible to discard some responses and to replace them with a local file. Item replace-response defines this replacement.

GIF, JPEG, and PNG images may be filtered according to the image dimensions. A local image is returned to the client instead. The image substitution is defined by filter-images. This feature can be used, for example, to filter advertisement banners, because thy often have known characteristic dimensions. Dimensions of GIF and PNG images are stored at fixed offset near the beginning of the respective files, but JPG dimensions may be far from the file beginning. Item jpeg-scan-sz restrict the size of initial part of JPEG files scanned for dimensions.

Http-proxy can filter data through an antivirus. Antivirus checking is defined by item antivirus which selects a top-level antivirus section. See antivirus(5) for details about configuration of virus checking.

Http-proxy provides HTML filtering. It is usually used to delete potentially dangerous parts of HTML data passed to client, e.g., scripts or Java applets. Features of the HTML filter are controlled by item html-filter, which selects a top-level html-filter section. See mod-html-filter(5) for details about configuration of HTML filtration.

HTTP request and response data can be processed and actions can be taken accordingly. Matching in request data is configured by item request-acl.request-body-match, response data matching is controlled by doc-acl.response-body-match. See data-matching(7) for detailed description of the data matching and processing feature.

The maximum size of the HTTP request body can be limited by setting request-acl.request-body-max-size.

If the request URI is categorized by clear-web-db, Bypass function can be enabled by clear-web-db-bypass. When accessing a matching page, the user gets an error page. By clicking a link on the page, access is enabled for a limited time to the blocked web server, or all servers belonging to the categories specified by clear-web-db-match.

Using CONNECT Method

HTTP method CONNECT is reserved for tunneling other protocols through http-proxy. It is usually utilized for SSL/TLS access to HTTPS servers. When a user sets its browser to use the proxy in the nontransparent mode, an HTTPS request causes a CONNECT request to be sent to the proxy. The proxy then creates a tunnel between the client and the server.

Note that data passed through the tunnel are encrypted and thus inaccessible to the proxy. After the tunnel is established, it is not possible to deny any HTTPS requests nor to perform data checking like HTML filtering or antivirus testing. It is therefore appropriate to limit the servers accessible via CONNECT.

In a transparent proxy configuration and HTTPS, the client does not use CONNECT, but it starts SSL/TLS immediately after establishing a TCP connection. Although it is possible to utilize http-proxy in this case (see the description of session-acl.simulate-connect below), it is usually easier to use tcp-proxy(8). An exception that requires http-proxy is when some HTTPS connections should be just passed via a TCP tunnel, but other should be decrypted by the proxy.

An alternative to a simple HTTPS tunneling is to use SSL/TLS decryption/encryption functionality of the proxy. A transparent http-proxy — which does not use CONNECT — can be configured to decrypt the connection from a client (by session-acl.ssl), process the encapsulated HTTP, and encrypt the connection to a server (by request-acl.ssl). A more complicated situation arises in the non-transparent mode. As mentioned above, the browser tries to establish a TCP tunnel through the proxy using the CONNECT method. If the request-acl contains item capture-connect, the proxy captures the CONNECT request, that is, it responds to the request as if the tunnel was established, but does not open the connection to the server. Instead, it restarts the session in transparent mode. End of the CONNECT request and session is logged, and a new session is started, which behaves as a transparent session to the server specified in the CONNECT. New session-acl and request-acl are selected that can, among other things, enable SSL/TLS decryption and encryption, in the same way as in a normal transparent http-proxy configuration. To be chosen for a new session emerged from a captured CONNECT request, a session-acl must contain item captured-connect. The session-acl selection can be based on the ACLs used for handling the CONNECT. Those ACLs are specified by items connect-session-acl and connect-request-acl.

If some connections should be decrypted and re-encrypted, but other ones are to be just passed, it is possible to set simulate-connect in a session-acl matching connection that will not be decrypted. This option behaves as if the data from the client were preceded by a CONNECT request to the destination address of the TCP connection from the client. That is, the proxy just establishes a tunnel and passes data unmodified between the client and the server. The proxy must learn the server address somehow, hence simulate-connect requires either a transparent proxy mode, or a plug-to item specifying the server address explicitly in an ACL.

The last option is to perform full inspection of the HTTPS. In this case, the http-proxy interrupts the initial phase of establishing the SSL connection from the client, it tries to contact the server and to get its certificate. If it fails, the connection to client will be reset. If the server is connected and its certificate is verified, the proxy generates a new certificate with all attributes (except some unwanted ones) from the original server's one, subscribes it by own certifiate authority and uses this new certificate for completing the connection to the client. If the original server certificate cannot be verified, then several options are available:

The new certificate is stored in the cache (a file in the /data/fake-cert directory) for later re-using. Correct certificates have names starting by the C letter followed by the certificate hash and distinguishing number. Wrong certificates (used in the pass case) have the F letter on the beginning, instead. See the ssl(5) manual page for further details.

Using FTP Scheme

When the client uses the proxy in the nontrasparent mode and the user requests data from an FTP server by entering a URL starting with ftp:, the client sends an HTTP request with that URL to the proxy. The proxy is then expected to fetch the document from an FTP server and return it as an HTTP response to the client. Http-proxy does not communicate directly with FTP servers. Instead, it asks ftp-proxy(8) for doing the work. Communication between http-proxy and ftp-proxy is done in a private protocol created specially for Kernun firewall. Parameters needed for connecting to ftp-proxy are specified by item ftp-proxy.

When the firewall works in the transparent mode, HTTP clients talk directly to FTP servers. Appropriately configured ftp-proxy is needed in such situation.

User Authentication

User authentication on proxies works in HTTP in a similar way as authentication on origin servers. The difference is in status codes (407 instead of 401) and headers (Proxy-Authenticate and Proxy-Authorization instead of WWW-Authenticate and Authorization). When the proxy requires authentication and a request does not contain valid credentials, the proxy replies with 407 and sends an authentication method and a realm to the client in header Proxy-Authenticate. The client then obtains user's credentials and repeats the request with them in header Proxy-Authorization. The credentials are sent automatically in all subsequent requests. Only the Basic, Kerberos (Negotiate), and NTLM authentication schemes are supported by http-proxy. The proxy can be configured for one or both of them. If both authentication schemes are enabled, a client can choose which scheme it will use. Typically, Kerberos/NTLM-capable web browsers will use Kerberos/NTLM, other browsers will use Basic.

Cookie Modification

The proxy can be cofigured to perform modification of cookies passed between a client and a server. The value of a cookie received from a server is replaced by a new value and passed to the client. If the client sends the cookie back to the server, the proxy restores its original value before passing it to the server.

This feature reduces exploitability of stolen cookies, especially session-identification cookies in various web applications. A cookie stolen from the client is useless outside the network protected by Kernun, because its value is not that expected by the server. Even inside the protected network, a stolen cookie has only limited potential of misuse, because after the proxy sends a cookie to a client, it accepts it back only from the same client IP address.

The proxy maintains a cookie table that is use for restoring modified values of cookies passed from a client to a server. To increase security, neither the modified cookie value passed to the client, nor the related record in the cookie table suffices for restoring the original cookie value. The two pieces of information must be put together in order to reverse the cookie modification operation.

Properties of the cookie table (file name, size, expiration, and cleaning rule) are set in section cookie-table. Rules for cookie modification are defined by items request-acl.modify-cookies. It is possible to modify only some cookies, selected by name, disable checking of client IP address by flag any-client, and decide whether cookie values sent by a client to a server and not found in the cookie table should be passed unchanged (flag keep-not-found) or replaced with an empty value. A request that uses a request-acl with item delete-cookies causes deleting all cookies related to a single IP address. Either the IP address of the requesting client, or the IP address contained (in standard textual notation) in the query part of the request URI, is used, according to flag ip-from-query.

Authentication Proxy (AProxy)

It is possible to configure http-proxy for providing access from an external network to some web server in the internal protected network. Often requirements in such configuration are encryption of the communication between the client and the proxy and using challenge-response authentication. Module AProxy of http-proxy provides this functionality. If a user is not authenticated, the proxy returns an authentication form instead of a normal response. When the user authenticates, the response for the original request is returned and further requests are processed normally until the user logs out or a timeout expires.

AProxy mode is switched on by item aproxy in session-acl. It is advisable to turn on SSL/TLS between clients and the proxy by item ssl in session-acl. Configuration section aproxy sets various AProxy parameters. Section auth defines AProxy authentication database. Username/password authentication is supported for both passwd and radius, challenge/response authentication may be used only with radius. User and group names obtained during AProxy authentication are matched against request-acl.aproxy-user condition.

The proxy identifies sessions belonging to authenticated users by cookies. It is necessary to choose a cookie-name so that it does not collide with cookies used by the origin server. The maximum number of simultaneously active user sessions is specified by max-aproxy-sessions. If insecure-cookie is not set, the client is asked not to send the session cookie across an unencrypted connection. It prevents possible revealing of the cookie when the user inadvertently enters http: instead https: into the browser.

Out of Band Authentication Server

Http-proxy is used also as an OOB authentication server, see auth(7). In this mode, the proxy manages the list of OOB authenticated users and provides the list to other proxies. OOB authentication server is turned on by a section aproxy containing item oob-auth. Parameters of the OOB authentication are set by a section oob-auth referenced by http-proxy.oob-auth-srv. OOB authentication uses either the html-form method (users authenticate themselves by filling the same form as in AProxy authentication) or the external method (the list of users is provided by an external program, e.g., ooba-samba(1), which passes it via HTTP to the authentication server).

Web Filter

The request URI can be processed by an external web filter. Interface to IBM Proventia Web Filter is implemented in the proxy. The web filter has a regularly updated database of web servers. It takes a request URI from http-proxy and assigns a set of categories to it (for example, pornography, games, lifestyle, criminal activities). Then it processes the categories together with client IP address and user name (if proxy authentication is enabled) and decides according to its ruleset whether the URI should be accepted or rejected. If the web filter accepts the URI, request processing continues in http-proxy. Otherwise, the proxy returns an error page to the client.

In the web filter configuration, ICAP Integration must be enabled (in Proxy Integration dialog of the management console). Also select User Profile Support in this dialog. In the Kernun configuration, section web-filter contains parameters of a connection to a web filter. Processing a request URI by the web filter is enabled by item request-acl.web-filter.

IBM Proventia Web Filter requires user names in the form domain\user. The http-proxy uses always domain name kernun. Therefore, user names in web filter configuration must be entered as kernun\user.

Program-Generated Responses

If item request-acl.program-response is set in the configuration, HTTP requests from clients are processed by an external program specified in this configuration item. A new instance of the program is started for each request. The complete HTTP request as received from the client is passed to the standard input of the program. The program must reply with a valid HTTP response written to its standard output and terminate. The proxy then sends the response back to the client. If the program does not terminate until a configured timeout or the request processing is interrupted before the program terminates, the proxy sends the SIGTERM signal to the running program.

In addition to the HTTP request on the standard input, the program is also provided with a set of environment variables:

Note that although this program interface resembles the CGI commonly used by WWW servers, it does not comply to the CGI definition in RFC 3875.

Running a Program at the End of Request

Item request-acl.request-end-program enables running an external program at the end of each request. The proxy does not wait for termination of the program. The program gets information about the request in the same set of environment variables as a program for generating responses described in the previous section. The suffix of the CONTEXT variable value corresponds to the request processing result as reported in the REQUEST-END log message.

Logging

As all other Kernun proxies, http-proxy generates many log messages during its operation. Meaning of the messages may be found in section 6 of the manual pages. Details about Kernun logging can be found in logging(7).

The proxy logs statistical messages about each client connection and each request. When a connection arrives, SESSION-START is logged. Then ACL message informs about the session ACL selected for this connection. Each request generates REQUEST-START (when the request line and headers are received from the client), ACL (selection of a request ACL), and REQUEST-END (at the end of request processing). Finally, SESSION-END is logged when the client connection is closed. If AProxy is enabled, login and logout of each user is reported as an APROXY-AUTH message.

Common Kernun Features

Http-proxy uses common Kernun mechanisms for listening on its sockets, accepting client connections, and managing its processes. It can also run in a chrooted environment and change its user identity upon startup. See also application(5), tcpserver(5), and tcpserver(7).

The proxy uses a common Kernun mechanism for network input/output. The configuration allows to specify several parameters like buffer sizes and timeouts, both for client and server connections. The parameters are set in configuration sections client-conn and server-conn. See netio(7) for details.

The proxy uses common Kernun mechanism for name resolving (see resolving(7) manual page).

Http-proxy uses common Kernun mechanism for runtime monitoring. For more detailed information, see monitoring(7).

Http-proxy uses common Kernun mechanism for traffic shaping. For more detailed information, see traffic-shaping(7).

common for all classes

acl-deny

Error response when the request is denied by request-acl or doc-acl.

The same set of substitution string is used also by the response classes clear-web-db-deny and by responses defined by request-acl.deny-msg and doc-acl.deny-msg.

clear-web-db-deny

Error response used when a request is denied by the Clear Web DataBase, that is, if a request-acl contains both items clear-web-db-match and deny.

bypass

The Clear Web DataBase Bypass activation page, returned if a request-acl contains both items clear-web-db-match and clear-web-db-bypass, and bypass has not been activated yet.

generic-error

Error response used when a soft error occurs. Description of the error is substituted for $8$.

cert-error

Error response used when a server presents an invalid certificate.

redirect

The response used when request-acl.rewrite contains a redirect. The redirection target URI is substituted for $8$.

ftp-response-put

Response for a PUT request with ftp: scheme. Result returned by ftp-proxy is substituted for $8$.

aproxy-password-form

AProxy form for entering user name and password.

aproxy-response-form

AProxy form which displays a challenge and asks for a response.

aproxy-logout

A page with information that the user has been logged out by AProxy.