  pthreads programming considerations? 
 The Question is:
 
I am designing an application in C++ and I would like to use DECthreads
(pthread_ interface).  After looking through the Guide to DECthreads, I have
a few questions:
 
1. Are there any limitations on the use of system services from within my
multithreaded app?
 
2. Can I issue $QIO calls that block?  If I do issue a blocking call, does
it block only the calling thread or does it block the entire process (i.e.,
if I block on a read from a mailbox, do a scanf(), or issue a UCX select())?
Does the thread scheduling algorithm affect the behaviour?
 
3. If (2) above doesn't block the entire process, how does VMS implement this
and does that implementation produce side-affects or other problems I need to
be aware of?
 
4. If I FORCEX() a process, do all of the thread cleanup routines get called?
 
 
 
 The Answer is:
Most of these questions have two sets of answers, depending on whether the
VMS kernel support for threads has been enabled or not.  The kernel support
is available on OpenVMS Alpha as of V7.0; the behavior on prior versions of
OpenVMS Alpha and on OpenVMS VAX is the same as the behavior on OpenVMS Alpha
V7 with the support disabled.  On OpenVMS Alpha V7 the support is enabled by
using a linker qualifier which controls whether "upcalls" and whether
multiple kernel execution contexts are available to the process.  The default
is that they are both disabled; note that the qualifier is only meaningful
when linking executable images, not shareable images.  (There is also a
SYSGEN parameter which can be used to control the maximum number of execution
contexts per process; setting it to zero disables upcalls and allows no
additional execution contexts to any process on the system.  By default, this
parameter is set to the number of processors on the system.)
 
 
In general, there are no limitations on the use of system services from
within a multithreaded application.  However, it's helpful to understand the
implications of calling any specific service as well as its interaction with
other threads.  The most important ramification is the effect of calling a
blocking service, as explained below.  However, there are other issues for
the application developer to consider, such as ensuring exclusive access to
output buffers, the use of event flags and the issues around sharing them
between threads, and other more specific problems (such as the fact that
taking out locks in different threads can cause the VMS Lock Manager to
incorrectly detect deadlocks).
 
You certainly can issue blocking system service calls from within a thread.
The effects on the calling thread as well as the other threads in the process
depends on whether the process has upcalls enabled or not.
 
With upcalls enabled, the VMS Exec notifies DECthreads that the executing
thread has blocked in a system service call.  (This is done by having the
Exec call "up" into a routine in the DECthreads scheduler, hence the term
"upcall".)  This allows DECthreads to deschedule the current thread and
immediately schedule another thread to run in its place.  When the block is
concluded, the Exec will issue another upcall to notify DECthreads that the
thread is now eligible to execute again, and DECthreads will schedule it at
the next point which is appropriate, based on the thread's scheduling policy
and priority.
 
Without upcalls, DECthreads has no way of determining that the current thread
is no longer actually executing when it is blocked in a system call.  As a
result, DECthreads will continue to schedule the blocked thread to "run",
according to the thread's scheduling policy and priority.  During the times
when the thread is scheduled, the process as a whole (i.e., all threads) will
be blocked.  However, with upcalls disabled, DECthreads uses a repeating
real-time timer AST to effect timeslicing among threads.  Thus, the process
only remains blocked for the remainder of the thread's quantum (or until it
is preempted by another thread), at which point DECthreads will regain
control and schedule another thread to run, if appropriate.  However, if
there is no reason for DECthreads to schedule another thread, e.g., because
the current thread has a priority which is higher than all other threads or
because the thread has a FIFO scheduling policy, then the thread will
continue to "run", and no other thread will get a chance to execute until the
blocking system service completes.
 
This mechanism (i.e., the one used in the absence of upcalls) works fairly
well for end-user applications, especially those which avoid heavy use of
priorities and FIFO scheduling.  These are typically cases in which periodic
short-term blocks don't pose a problem.  However, the mechanism doesn't work
out as well in performance-critical applications or in certain specialized
coding situtations.  The issues in the former should be obvious. If an
application maintains a hundred network connections with a thread blocked on
a read on each connection, then the latencies between I/O completion and
resumption of execution could be in the tens of seconds!  Likewise, if a
thread which is executing in such a process reaches the end of its quantum,
it would be a similar period of time before it was assigned to run again.
The other problems arise from the fact that the timeslicing depends on a
User-mode AST.  That is, if the system service blocks in a more privileged
mode, such as Exec- or Kernel-mode, then it is not subject to interruption by
the timeslice AST.  (Most services, such as $QIOW, do their waiting in
"caller's mode" -- i.e., User-mode in the typical case -- but a very few do
block in inner mode.)  Furthermore, if an application, run-time library
routine, or system service disables ASTs, it likewise will block the entire
process for the duration of the blocking system service call.  (Again,
basically no Digital-supplied system services disable ASTs, and almost none
of the run-time library routines do, but it's not uncommon for application
code to do so, and there have been several reports of routines for socket I/O
blocking the process with ASTs disabled.)
 
For these and similar reasons, with V7.0 VMS provides kernel support for
threads on Alpha, including upcalls.  With the addition of upcalls, the Exec
can notify DECthreads immediately when a thread is blocked or no longer
blocked in a system service call, and this frees us from the issues above.
Furthermore, it allows DECthreads to use a timeslice interval which is based
on CPU-time instead of real-time which requires less overhead and produces
much fairer distribution of execution on a busy system.  And, it frees
DECthreads to a large extent from problems resulting from the application
disabling ASTs.
 
In addition, it also provides for much more robust operation of services like
$EXIT and $FORCEX in a multithreaded process.  Exit handling presents special
challenges in a multithreaded process, because the exit handler routines may
need to synchronize with other threads in order to gain access to resources
which need to be cleaned up.  Without upcalls, a call to $EXIT made from
within a thread might invoke a routine which blocks waiting for a resource
already held by the calling thread, resulting in a deadlock situation which
makes the process hang instead of exiting.  $FORCEX is even more
trouble-prone, since in this case the exit handlers are invoked
asynchronously in an arbitrary thread, so there is no possibility of
preparation, such as releasing resources, prior to the invocation of the exit
handler routines.  However, with upcalls enabled, exit handler routines are
executed in a separate thread created expressly for the purpose; a call to
$EXIT or $FORCEX simply makes this thread eligible to run.  Thus, if an exit
handler routine requires a mutex, the executing thread is free to block
without risking a deadlock.  Furthermore, when a thread calls $EXIT with
upcalls enabled, it results in an implicit call to pthread_exit(), which
unwinds the calling thread's stack allowing any frames with TRY blocks,
pthread cleanup handlers, or VMS condition handlers to clean up and release
any resources which the thread is holding (to allow the exit handlers to
proceed without deadlocking).
 
Independent of upcalls, it is important that the application developer
provide for an orderly application shutdown.  Just as a thread must not be
allowed to call a subsystem before that subsystem is initialized, it is up to
the application developer to ensure that no threads call a subsystem after it
has been shut down by its exit handler.  (The best example of this is calls
to printf() during or after the execution of the C RTL's exit handler that
flushes and closes all open stdio files.)  One approach is for each subsystem
to declare an exit handler during its initialization which shuts down and
joins with all the threads that the subsystem created:  since the exit
handlers are run in the reverse order from that in which they were declared,
this ensures that each thread can clean up properly, including calling other
subsystems and that when a given subsystem is shut down all of its consumers
are already shut down.
 
Note that thread clean-up is only done in threads which actually terminate
before the process is run down (rundown happens after the last exit handler
routine completes).  At process rundown, all unterminated threads and the
rest of P0-space memory is summarily destroyed.  Thus, if an application is
depending on thread clean-up to maintain external invariants, it must ensure
that it has an exit handler which joins with the critical threads, to ensure
that they have been properly shut down prior to process rundown.
