critical impact:

setsid - controlling terminals acquired differently (CrCh260)

Problem description

Controlling terminals are acquired differently on Tru64 UNIX and HP-UX. As a result, applications that call setsid might need to make code changes.

The setsid function creates a new session. The calling process becomes the session leader of the session and the process leader of a new process group, and loses any controlling terminal.

On Tru64 UNIX, a process can only acquire a new controlling terminal by calling ioctl with the TIOCSCTTY request. On HP-UX, that request is not available. Instead, a process acquires a new controlling terminal automatically the first time it opens a terminal device that is not already associated with a session, providing the open call does not specify the O_NOCTTY flag. (The open function sets O_NOCTTY by default on Tru64 UNIX, but not on HP-UX.)

These differences can cause the following problems for applications migrating to HP-UX:

• Applications using TIOCSCTTY need to make code changes because the TIOCSCTTY ioctl request is not available.

• Daemons opening /dev/console to log error or status messages may inadvertently acquire it as a controlling terminal if they are not coded properly.

Identifiers

setsid

See also

• Background information on kernel impacts
• setsid(2)
• tty(7)

Solution description

Modify your application.

If the application uses TIOCSCTTY to acquire a controlling terminal, enclose the ioctl request within #ifdef TIOCSCTTY and make sure the corresponding file descriptor is opened after the call to setsid. Do not open the file descriptor before the call to setsid.

To prevent a daemon from inadvertently acquiring a controlling terminal, call fork after calling setsid. This lets the parent exit and the child continue. (This is in addition to the fork that is normally done before calling setsid.) The child will then no longer be a session leader and thus no longer able to acquire a controlling terminal. An alternate solution is to use the O_NOCTTY flag when opening terminal devices.

New behavior

/* example 1: acquire controlling terminal */

(void) setsid();
fd = open(tty, O_RDWR, 0);	/* HP-UX acquires ctty here */
#ifdef TIOCSCTTY
(void) ioctl(fd, TIOCSCTTY, 0); /* Tru64 UNIX acquires ctty here */
#endif

/* example 2: prevent acquiring controlling terminal */

(void) setsid();
if (fork())
        exit(0);

See also

• setsid(2)
• termio(7)

Problem summary