| Related sites for http://www.feyrer.de/NetBSD/gmcgarry/ |
| Advanced_Forum CGI application that implements discussions on a website. | | Techknowledge_Training Headquartered in Maryland, with z/OS-based scheduled public mainframe courses in U.S. cities. | | WorkSpot Free and rental software over the Internet, as well as traditional file storage. This content is available through Linux, HTML, and Palm VII interfaces. | | Free_Documents_for_Smalltalk Links to several online tutorials, books; VisualWorks, OO analysis and design, Jun OO Smalltalk application framework and 3D graphics library; many documents and programs distributed under GPL license | | paxPascal Implements a subset of Object Pascal and extends it with a few extra features, including namespaces, dynamic record and array types, inheritance for record types. It's one of the languages of the paxS | | Envision_Presentations Offers a full set of presentation tools for Realtors. | | buzz_dev_files Examples of source code. | | Meshbank Free 3D models in .obj format. | | Praktical_com_-_Ultra_Tech_Delphi_Components_ Organic Shape Pack to make your applications skinable like WinAmp. And PanoMax Panorama Component to render real-time views of panoramic (360 degrees) images, giving a Virtual Reality-like immersion f | | Remunerate Australian company offering employee salary and incentive compensation review software. | | FuturePlus_Systems Power tools for bus analysis. Solutions partner of Hewlett-Packard's. | | Toxicglitter Webring for people who are 20 something. | | RFC_3127 Authentication, Authorization, and Accounting: Protocol Evaluation. D. Mitton, M. St.Johns, S. Barkley, D. Nelson, B. Patil, M. Stevens, B. Wolff. June 2001. | | My_Favorites Accelerate your Web surfing by organizing and launching your IE favorites with one click, and use them to customize your Web Browser for each favorite Website. [Shareware] | | Programmershelp Various Visual Basic code snippets. | | BreakMyGentoo Provides additional packages that are not provided in the main distribution and a guide about how to set the CFLAGS for compiling. | | Turning_up_the_heat_on_Web_privacy News and Technology - CNETAsia - When Microsoft introduced version 6 of its Internet Explorer browser last year, many Webmasters were puzzled to find that their cookies were being blocked in increasin | | Navigator_Utilities Eases navigating through complex spreadsheets. Provides company and purchasing information, free demo and a presentation of the product. | | Quirino,_Tim_-_Shiftcore Offers web design, and graphic/print design services. Located in Flanders, New Jersey, United States. | | OpenSUSE_us Unofficial forums for the community-driven version of this distribution. Topics include support for installation, networking, and hardware. |
|
Benchmark Comparison of NetBSD 2.0 and FreeBSD 5.3This article is all from Gregory, I'm only providing the webspace! - HubertBenchmark Comparison of NetBSD 2.0 and FreeBSD 5.3Gregory McGarryg.mcgarry@ieee.orgAbstract:With the recent releases of NetBSD 2.0 and FreeBSD 5.3 operatingsystem, many new and exciting features have been implemented. Bothcriticism and commendation on performance, reliability and scalabilityhave been directed towards these releases.This paper presents a suite of benchmarks and results for comparingthe performance of these operating systems. The benchmarks targetcore operating system functionality, server scalability and threadimplementation. These benchmarks are useful server-based criteria fordemanding applications such as loaded webservers, databases, andvoice-over-IP (VoIP) media relays. The results indicate that NetBSDhas surpassed FreeBSD in performance on nearly every benchmark and ispoised to grab the title of the best operating system for the serverenvironment.IntroductionThe FreeBSD and NetBSD operating systems are complete Unix-likeoperating systems sharing a common BSD Unix lineage. Although bothoperating systems share a similar development style and source code,each project has traditionally had very different objectives. Askalmost anybody familiar with at least one of the projects as to whichis the better operating system, and the reply will usually recommendFreeBSD for servers and desktop systems, and NetBSD for obscurehardware. FreeBSD has historically been clean, fast, reliable andscalable and NetBSD is known to support fifty-four different systemarchitectures.However, in recent years, the traditional arguments for choosing oneoperating system over the other has waned. FreeBSD now supports sixpopular system architectures and NetBSD recently set Internetland-speed records.Long criticized for their slow release schedules, the NetBSD Projectannounced in December 2004 the release of NetBSD 2.0. NetBSD 2.0represents more than two years of development over the last majorrelease, and includes significant developments in the areas ofsymmetrical multiprocessing (SMP) and high-performance POSIX threads.Despite the long delay in delivery, the release has been warmlyaccepted by the industry.One month earlier, the FreeBSD Project announced the release ofFreeBSD 5.3; the first ``stable'' release of the FreeBSD-5 developmentbranch. In contrast with NetBSD 2.0, FreeBSD 5.3 has received abarrage of criticism of its performance, scalability andreliability[1]. Indeed, the FreeBSD-5 branch has beenplagued with problems from the very beginning. These problems haveprimarily centered round an ambitious new scheduler, a complicatedthreading model and fine-grained SMP architecture. All these problemshave left most FreeBSD installations stuck with the aging FreeBSD-4branch.With simultaneous releases of NetBSD 2.0 and FreeBSD 5.3, the industryis now returning to the original questions: Which is thebetter operating system? For servers? For embedded? However, thistime, the traditional response is not being used.In the remainder of this paper, benchmark comparisons arepresented to compare the underlying performance of the operatingsystems.BenchmarksThe responsibility of the operating system is to manage resources andprovide services to applications to access these resources. Indemanding environments, significant load will be placed on theseresources and the performance of the operating systems under this loadis of specific interest. Some of the benchmarks measure theefficiency of the operating system for frequent, time-sensitivefunctionality. Other benchmarks seek to identify the scalability ofthe operating system during network operations[2].Three categories of benchmarks have been used. The first categorymeasures core operating system functionality:system-call overheadcontext-switch timeprocess creation timeprocess termination timeprogram load timeThe second category measures the scalability of the operating systemunder application and network load:process creation time for increasing loadprocess termination time for increasing loadmemory-mapped file setup timememory-mapped file access timesocket creation timelatency to bind an address to a socketThe last category measures the overhead of the native threading model:thread creation time for increasing loadthread lifecycle overheaduncontested mutex access overheadcondition variable access timethread context-switch timeThe benchmark hardware was an Asus P4-800SE mainboard, Intel 3GHz P4processor (1MB L2 cache) and 1GB RAM. Both NetBSD 2.0 and FreeBSD 5.3default installations were used for the benchmarks. No custom kernelswere used and no kernel tuning beyond the default installation wasperformed.Benchmark ResultsThis section presents the benchmark results for the three benchmarkcategories.Core operating system functionalityThe primary resource managed by the operating system is the CPU.Management of process access to the CPU is managed by the kernelscheduler using priority-based, time-slice allocation. On asingle-CPU machine, the appearance of parallelism is achieved byquickly switching between process contexts. There is CPU overheadassociated with this operation. It is important for the operatingsystem to minimize this overhead, particularly during high systemload.Through general use of the operating system, many processes will becreated, executed and terminated. This lifecycle is an integral partof the operating system and the efficiency of the operating system toperform it is important. All processes in BSD are created using thetraditional fork/exec Unix process-creation model.This model was part of the original Unix design, and is stillimplemented in virtually every version of Unix available today.The fork and exec operations are system calls. Asystem call is the interface between the user-level application andthe kernel. The fork system call creates a new process. Thenewly created process gets a unique process identification and is achild of the process that has called fork. The calling process is theparent. The exec system call overlays the process addressspace with data from an executable file. A process is terminated withthe _exit system call. It is generally invoked indirectlyusing exit() in the standard C library.The benchmarks developed in this section measure these importantoperating system functionality. The results are shown intable 1.Table 1:Performance comparison of core operating system functionality.benchmark ( s) NetBSD FreeBSD system-call overhead 0.368 0.393 context-switch 2.64 3.45 process creation (dynamic) 146 168 process creation (static) 63 80 process termination (dynamic) 43 64 process termination (static) 32 36 program load (dynamic) 463 1424 program load (static) 96 266 System-call overheadThe system-call overhead benchmark measures the time for the operatingsystem to switch from unprivileged user mode to privileged kernel modeand returning to unprivileged user mode. This metric is performed bymeasuring the time to execute the getpid system call, whichis the simplest system call available in the operating system.The results in Table 1 shows that NetBSD 2.0 marginallyout-performs FreeBSD 5.3.Context-switch timeThe context-switch time benchmark measures the time for the operatingsystem to switch from the execution context of a process to another.A context switch requires the kernel to save the address space and CPUregisters of the current process and load them for the next process.The kernel must also manage coherency of the CPU caches. This metricis performed by creating two processes connected with a bi-directionalpipe. A one-byte token is passed alternately between the twoprocesses, causing a context switch to each process to services accessto the pipe.The results in Table 1 shows that NetBSD 2.0 marginallyoutperforms FreeBSD 5.3.Process lifecycleThe process creation time measures the execution time of thefork system call. There are several optimizations which theoperating system will perform to optimize the creation of the newprocesses. During a fork system call, the kernel will sharethe parent address space read-only with the child. A newprocess identification is allocated to the child process in additionto other data structures used for process-specific accounting. If thechild subsequently writes to the address space it shares with itsparent, a copy-on-write of the altered page is made into thechild address space. However, if the child process terminatesimmediately, or invokes the exec system call, theaddress-space is returned unchanged to the parent without performing acomplete address-space copy. This metric is performed by creating apipe between the parent and child processes. The parent measures thetime for the child process to be created when it receives a one-bytetoken sent on the pipe from the child process.Process termination time is equally important. The address space mustbe reclaimed in addition to other data structures used forprocess-specific accounting. This metric is performed by the parentsending the child process a SIGTERM signal and invoking thewait4 system call to wait for child termination.The execution times of the fork and exec systemcalls are also sensitive to the address space layout of the process.In particular, dynamic-linked executables using relocatable objectsproduce sparse address spaces in comparison to static-linkedexecutables. Memory management is impacted by the sparse addressspace. The benchmarks have been developed to compare the metrics fordynamic-linked and static-linked executables.The results in Table 1 indicate that NetBSD 2.0marginally outperforms FreeBSD 5.3. The performance difference fordynamic-linked executables is particularly noticeable.Scalability benchmarksScalability refers to the ability of the operating system to performoperations during increased demand for resources. Scalability isparticularly important on systems with high application and networkload. For example, a system running a demanding network or databaseserver may require significant network sockets and fork many workerchild processes. The performance of the operating system as thesystem load increases is an important criteria for system architecture.Forking new processesContinuing from the previous section, the first benchmark considersthe process creation and termination times as the number of systemprocesses increases. This benchmark measures the architectural designof the scheduler. Results for process creation and termination timesfor dynamic-linked executables are presented inFigure 1 andFigure 2 respectively. The results forstatic-linked executables are shown inFigure 3 andFigure 4 respectively.Figure 1:Process creation times for increasing number of systemprocesses with a dynamic-linked executable. Figure 2:Process termination times for increasing number of systemprocesses with a dynamic-linked executable. Figure 3:Process creation times for increasing numberof system processes with a static-linked executable. Figure 4:Process termination times for increasing numberof system processes with a static-linked executable. The plot for process termination may be difficult to interpret. Thebenchmarks starts with 3000 system process and measures the time toterminate each process as the number of system processes is reduced.These results indicate the NetBSD kernel has very efficient datastructures for managing system processes to permit constantaccess-time to allocate new process resources, such as a processidentification, and to locate the process for termination. For thisbenchmark, NetBSD has excellent scalability, since the time to performprocess creation and termination is not affected by the number ofsystem processes.The FreeBSD kernel has an access time which scales linearly with thenumber of system processes. There are also many occasions when theaccess-time is very fast, resembling a constant access time. Anexplanation for this result may be that an optimization within theFreeBSD kernel is performed to arrange access to resourcesappropriately to minimize the access time. However, there are alsomany occasions when the access-time is half as fast as the nominalvalue. Perhaps the optimization does not work well for everyworkload?Memory-mapped filesThe mechanism for mapping shared libraries into a process addressspace is the mmap system call. This system call permits aprocess to map data of arbitrary size from a file into its addressspace. The mechanism is also used by some databases, web servers andproxy servers to map files into memory rather than reading the filecontents into the system buffer cache.To support the mmap system call, the operating system mustmaintain data structures for system-wide memory accounting and datastructures for the process-specific file-mapped data. A benchmark isused to measure the performance of these datastructures[2].The benchmark maps many 4KB windows from a 200MB file, each offset by4KB from the previous window. A common optimization for the operatingsystem is to ``lazily'' map the data into the address-space on thefirst access, rather than mapping it at the time of the mmapsystem call. The benchmark measures the time to invoke the initialmmap system call, shown in Figure 5, andthe time to access the first byte of the mapped window, shown inFigure 6.Figure 5: Comparison of times to map a 4KB window from a file intoprocess address space for increasing number of mappings. Figure 6:Comparison of access times for a 4KB memory-mapped window forincreasing number of mappings. The results show excellent scalability for both NetBSD and FreeBSD.Both operating systems permit constant access-time for mapped-memoryfiles with increasing load. FreeBSD has the peculiar behavior thatthe memory mapping time is clearly distributed around two values.For the initial mmap system call, NetBSD has 113% improvedperformance over FreeBSD. For accessing the memory-mapped file,FreeBSD has 116% improved performance over NetBSD. For frequentmapping operations which are not accessed, NetBSD will show betterperformance. However, for the general case with infrequent mappingoperations and frequent accesses, FreeBSD will show betterperformance. The total of both benchmarks indicate that for a singlemapping and subsequent access, FreeBSD shows a 38% performanceimprovement over NetBSD.Socket creation scalabilitySockets are the basis for communication in BSD operating systems.They can be used for inter-process communication and networkcommunication.A socket is created using the socket system call. The systemcall returns a file descriptor which permits the traditional fileoperations to be used for communication. Benchmarking thesocket system call measures the time the kernel takes toallocate the necessary data structures and the time to find the lowestunused file descriptor for the process. This latter task becomes moredifficult for a large number of open file descriptors and affectsscalability.The results for socket creation for increasing number of allocatedsockets are shown in Figure 7. The results show thatsocket creation time for both NetBSD and FreeBSD increases linearlywith the number of allocated sockets. The rate increases slightlymore for FreeBSD than NetBSD, and NetBSD produces faster allocationtimes. Neither NetBSD nor FreeBSD shows scalability problems.Figure 7:Socket creation time for increasing number of allocatedsockets. Binding addresses to socketsWhen a socket is created with the socket system call, itexists for an address family, but does not have an accessible addressassociated with it. The bind system call assigns an addressto the socket.This benchmark measures the time for the kernel to allocate thenecessary resources to assign an unused TCP port to the socket. Thebenchmark is not important for scalable web servers, however it isimportant for voice-over-IP (VoIP) proxy servers and media relays.The results for the socket binding benchmark are shown inFigure 8. The results show that both NetBSD and FreeBSDscale linearly with the number of bound sockets. For a small numberof bound sockets, NetBSD has the better latency than FreeBSD.However, the linear gradient for NetBSD is significantly worse thanFreeBSD. This result indicates the FreeBSD scales better for bindingaddresses to sockets.Figure 8:Socket bind time for increasing number of bound sockets. Thread BenchmarksNetBSD 2.0 is the first release to support the new threads systembased on scheduler activations[3]. It includes theimplementation of a POSIX-compliant threads library that uses thescheduler activations interface. The library includes manyoptimizations to attain impressive performance[4].Thread creation benchmarkThe importance of the process lifecycle was previously mentioned; fromprocess creation, through process scheduling, to process termination.Similarly, threads go through the same lifecycle and the performanceof the thread implementation to manage this lifecycle is important.For many applications, the demands on the thread implementation aremore onerous than the demands on the kernel scheduler.The times to create a new thread for increasing number of threads isshown in Figure 9. This result shows that theFreeBSD thread implementation creates new threads with constant time.This result indicates that FreeBSD has good scalability. The creationof the first thread in NetBSD has significant latency. This resultoccurs because the NetBSD threads implementation defers theinitialization of the threads library to the creation of the firstthread rather than doing the initialization when the threads libraryis loaded.For less than 250 threads, the time to create a thread is better inNetBSD than FreeBSD. For more than 250 threads, the thread creationtime increases as the number of threads increases. Of particularconcern, the relationship is not linear for the number of threads.Although one thousand threads is ample for most multi-threadedapplications, the poor scalability may be a problem for someapplications.Figure 9:Thread creation time. Thread lifecycle, condition variables and mutexesAnother benchmark is to measure the complete lifecycle of a thread, bycreating, joining and terminating a thread. The processing time ispresented in Table 2.Threaded applications also make extensive use of mutexes and conditionvariables to serialize access to shared data. Mutexes are commonlyused and in many cases their access is uncontested. A benchmark whichmeasures the time to acquire an uncontested mutex is shown inTable 2. Another benchmark measures the time for athread waiting on a condition variable to respond. The benchmark usesten worker threads and a master thread. The worker threadsconditionally wait on a variable that the master thread sets. Whenthe variable is set, a signal is broadcast to wake any worker thread.The woken worker thread clears the condition variable, signals themaster to set the condition variable, and immediately waits on thecondition variable. The results of the benchmark are shown inTable 2.Table 2:Benchmark comparisons for thread operations.benchmark (s) NetBSD FreeBSD thread lifecycle 2.35 4.33 uncontested mutex 0.0372 0.282 condition variable wakeup 1.21 4.86 The results of these benchmarks for the basic POSIX thread primitivesclearly shows that the NetBSD thread implementation contains manyimpressive optimizations. The threads implementation framework basedon scheduler activations permits low-overhead, efficient allocationof CPU resources to the threads[3]. The effect ofrestartable atomic sequences for the mutex implementation is alsosignificant[4].Thread context-switch timeThe reasons for optimizing process context switches equally appliesfor optimizing thread context switches. The ``ping-pong'' benchmarkmeasures thread context-switch time for different number of threads.The benchmark was devised to test the quality of Solaris threadslibrary implementations[5].The benchmark consists of a pair of threads in lock-stepsynchronization, resembling the game of ``ping-pong''. For eachiteration, each thread is alternately blocked and unblocked by theother. The game continues until a specified number of iterations hasbeen completed by each thread. The game also permits the playing ofmultiple concurrent games within a single process, testing theperformance when the thread scheduler is under load.Three experiments are considered:one game, 2 players, 1000000 iterations, default stack sizefour games, 8 players, 1000000 iterations, default stack size500 games, 1000 players, 100 iterations, 32KB stack sizeThe number of mutex operations and the number of context switchesshould be twice the number of iterations. The times to create thethreads and play the game are shown in Table 3.Table 3:Result of the ``ping-pong'' benchmark.benchmark (s) NetBSD FreeBSD 1 thread creation time 374 215 game completion time 954 660 3 139 256 2 thread creation time 517 446 game completion time 5 049 811 13 062 477 3 thread creation time 23 535 41 154 game completion time 140 551 275 145 Experiment 1 shows that the creation time for the two processesfavors the FreeBSD implementation over the NetBSD implementation.This result agrees with the results shown inFigure 9. The creation of the first threadincurs a significant penalty on NetBSD. However, the time to completethe game is significantly higher for FreeBSD over NetBSD. This is dueto increased latency in the thread lifecycle and the much longer mutexacquire time for FreeBSD over NetBSD.Experiment 2 shows similar results to Experiment 1. For four threads,the overhead of the initial thread creation in NetBSD is not assignificant. The improved processing performance of NetBSD overFreeBSD is similar for both experiments.The default thread stack size in NetBSD is 2MB. To create a largenumber of threads, the stack size can be reduced using thePTHREAD_STACKSIZE environment variable. In experiment 3, the stacksize is reduced to 32KB. For 500 threads, the faster thread creationimplementation on NetBSD outperforms the FreeBSD implementation. For500 simultaneous games, the margin of improvement of NetBSD overFreeBSD is noticeable, but less significant as the other experiments.ConclusionsThis paper has presented a suite of benchmarks and results forcomparing the performance of NetBSD 2.0 and FreeBSD 5.3 in the areasof core operating system functionality, network scalability and threadperformance.The results clearly indicate that recent architectural decisions inthe NetBSD operating system have closed the performance gap betweenNetBSD and FreeBSD. In fact, NetBSD has surpassed FreeBSD inperformance in the areas investigated in this paper. Significantperformance improvements are obviously visible in the threadimplementation.Microbenchmarks are not always the best indicators to make judgmentson the overall performance of one operating system over another.However, they are useful to infer an understanding of thearchitectural decisions that go into building an operating system.For many applications, the results presented in the paper may neveraffect performance. For others, the scalability of the operatingsystem may simply not permit the application to run suitably.There are many other interesting developments in NetBSD 2.0 andFreeBSD 5.3 that deserve to be compared. Although NetBSD 2.0 hasoutperformed FreeBSD 5.3 in most of the benchmarks presented here,FreeBSD 5.3 has made significant developments with its symmetricmultiprocessor (SMP) architecture, particularly in the area ofscalability with fine-grained locking. NetBSD 2.0 continues to use asingle lock to serialize access to kernel mode. Additionally, theperformance of the thread implementation on multiprocessor systems,where thread concurrency can be achieved, would be worthinvestigating. Benchmarks for these areas are the objective of futureresearch.AcknowledgmentsThe scalability benchmarks were supplied by F. vonLeitner[2]. The threads benchmarks were provided by theGelato project at the University of New South Wales, Australia(http://www.gelato.unsw.edu.au/IA64wiki/NPTLbenchmarks). The``ping-pong'' benchmark was provided by SunMicrosystems[5]. Source code for the other benchmarks canbe found atftp://ftp.netbsd.org/pub/NetBSD/misc/gmcgarry/bench/. Bibliography1J. Matzan, ``FreeBSD 5.3 is ``stable'' but not production-ready'', http://www.newsforge.com/article.pl?sid=04/12/14/1518217, December 2004.2F. von Leitner, ``Benchmarking BSD and Linux'', http://bulk.fefe.de/scalability/, October 2003.3 N. Williams, ``An implemenation of scheuler activations on theNetBSD operating system'', In Proceedings of the 2002 UsenixAnnual Technical Conference, 2002.4G. McGarry, ``An implementation of Restartable Atomic Sequences on the NetBSD operating system'' USENIX Annual Technical Conference, June 2003.5Sun Microsystems, ``Multithreading in the Solaris OperatingEnvironment'', A Technical White Paper, 20022005-01-03Access count:60357 |
|
