The Design of Software (CLOSED)

A public forum for discussing the design of software, from the user interface to the code architecture. Now closed.

The "Design of Software" discussion group has been merged with the main Joel on Software discussion group.

The archives will remain online indefinitely.

Concurrent Programming - Learning Advice

Ok.  Most of you have probably read Eric Sink's article on Concurrent Programming:  http://software.ericsink.com/entries/LarryO.html

And there have been many other folks who are talking about the why we (developers) need to start learning about concurrent programming and using it.

So... where do I start?  I primarily code in .NET (VB and C#).  I have started playing with the BackgroundWorker in a test project to see how it works, but I don't know where to truly begin with concurrent programming.

Many of the applications out there do not need concurrent ability.  But there are many that will.

Any books, blogs, tutorials, etc that you can recommend (or anything else of interest).
Eric D. Burdo Send private email
Monday, August 21, 2006
 
 
I'm about halfway through this book:

Java Concurrency in Practice
http://www.amazon.com/gp/product/0321349601/

It's Java but still may be useful for you. I think it does a great job of describing common misunderstandings in developing threadsafe software and shows the better alternative, doing so in a very readable style. FWIW, the latter half of the book seems to be targeted more toward the new concurrency features in Java 5.
Chris Winters Send private email
Monday, August 21, 2006
 
 
Two excellent tutorials on multithreading in .NET. They are both spoken in C# but the principles are generic:

Jon Skeet: http://www.skeet.org.uk/csharp/threads/

Joseph Albahari: http://www.albahari.com/threading/
Larry Lard Send private email
Monday, August 21, 2006
 
 
How much of this is WAY offbase?

Most machines have plenty of processes running around in them to soak up available processors, cores, or hyperthreads.  Once these architectures improve to the point where SMP machines are truly symmetric (i.e. we get away from cache and other issues forcing thread-processor affinity) we'll see the available resources round-robined or otherwise scheduled much more efficiently too.

Most servers seem to idle a LOT, leading to running multiple VMs in "bigger" boxes as a trend.  What's THAT do to the idea of splattering a program across a lot of threads and processes on purpose, in the hope of soaking up all of the processor resources?

I'm not sure the deep thinkers on this topic have thought deeply enough yet.
S. Nationalie
Monday, August 21, 2006
 
 
S. Nationalie - good question.  That's part of why I asked this.

Is this something I really need to be working on?

I see two sides to the concurrent programming.

Side 1: Using threads to make your app respond quickly (such as using BackgroundWorker in C#).

Side 2: Running multiple threads that communicate to break up large, complex routines into smaller ones.  Or something similar.

Side 1 is good for just about everyone to at least look into.  A fast responding app keeps your customers happy.  Side 2 isn't as useful for many apps (outside of the server realm).
Eric D. Burdo Send private email
Monday, August 21, 2006
 
 
See http://www.ddj.org/191800187

I think these guys are offbase too.  This was mainstream tech in the mainframe world from the mid-60s on and even earlier in specialized environments.

Minis like those from Tandem, et al. were also awash in this forever.

Of course that doesn't mean most programmers are or ever were all that good at it.  That's why there used to be a hierarchy among developers, from grunt coders through Systems programmers.
B. Afraid
Monday, August 21, 2006
 
 
"How much of this is WAY offbase?

Most machines have plenty of processes running around in them to soak up available processors, cores, or hyperthreads.  Once these architectures improve to the point where SMP machines are truly symmetric (i.e. we get away from cache and other issues forcing thread-processor affinity) we'll see the available resources round-robined or otherwise scheduled much more efficiently too."

It seems to me that the main concern regarding the need for concurrent programming is for individual applications, which almost universally still run as a single process.  This means that the they can't be spread across multiple processors.  So to get better performance from individual apps, developers are going to have to make use of threading, i.e., spawn new processes as part of an individual application.

"Most servers seem to idle a LOT, leading to running multiple VMs in "bigger" boxes as a trend.  What's THAT do to the idea of splattering a program across a lot of threads and processes on purpose, in the hope of soaking up all of the processor resources?"

Nothing.  The trend toward VM's has nothing to do with the fact that servers idle a lot.  If that were all there was to it, then people would just install multiple servers on the same machine (with no VM's). 

Gains from multithreading are not something you go to a VM for, so far as I understand them.  VM's are used to run an OS environment with pieces that might conflict with other apps on Host OS, to run different guest OS's for testing purposes, simplify configuration, gain portability of VM's across machines, and other reasons.  I don't think threading has anything to do with it.
Herbert Sitz Send private email
Monday, August 21, 2006
 
 
I've seen it a lot in scientific computing, mostly (but not exclusively) in academic settings where you have access to huge computing clusters, or, at a lower level, in embedded systems where pipelining your algorithm is important. I'm not sure how meaningful that will be for mainstream applications with little heavy computing.
BossyKena Send private email
Monday, August 21, 2006
 
 
> I'm not sure how meaningful that will be for mainstream applications with little heavy computing.

I think it will be more valuable in the grid context where you are process great torrents of data.
son of parnas
Monday, August 21, 2006
 
 
"Gains from multithreading are not something you go to a VM for, so far as I understand them.  VM's are used to run an OS environment with pieces that might conflict with other apps on Host OS, to run different guest OS's for testing purposes, simplify configuration, gain portability of VM's across machines, and other reasons.  I don't think threading has anything to do with it."

I *suspect* that the concern expressed was how virtualization may defeat attempts to use resources better through gratuitious multithreading.  I.e. the application developer doesn't usually have a lot of say about the physical (or virtual) infrastructure the application runs on.  So perhaps taking "concurrent programming" as a paradigm for everything will result in even poorer utilization of hardware.

Do most VM products run individual VMs locked to a processor on conventional x86 SMP hardware?
Artad Gobeski
Monday, August 21, 2006
 
 
Artad -- Thanks, you're correct that I misunderstood the concern.  Below is interesting snippet from VMWare Knowledgebase, simple answer, I think, is that each VM can be scheduled by host OS to run on a different processor:

"Question: Can I run VMware products on a multi-processor host machine?

Answer: The VMware software will run on Symmetric Multi-Processor (SMP) systems, also technically referred to as Multi-Processor Specification (MPS) systems. However, the environment provided within each VMware virtual machine is a Uni-Processor (UP) system. Multiple concurrent VMware virtual machines will make use of the multiple processors in a system.

The VMware software requires at least a Linux 2.2.x kernel to run on an SMP system and will fail with an error message on SMP systems running 2.0.x or 2.1.x kernels. With VMware products for Windows NT/2000/XP, the VMware software supports SMP kernels. The host OS will run its tasks on multiple processors dynamically (regardless of whether apps or one, two or more VMs are running at the same time). Keep in mind though that each VM runs as a single process on the host OS, so if you run a single VM on a dual processor machine, the performance will be only slightly higher then if you ran the same single VM on a single processor system. However, if you run two VMs concurrently, the dual-processor machine will yield higher performance since the host OS will assign each of the two main VM processes to each processor.

At this time, ESX Server 2.x virtual machines have the option of running in SMP mode, with the Virtual SMP add-on available from VMware:

www.vmware.com/products/server/vsmp_features.html

VMware has plans to support SMP inside a virtual machine environment with our other products; however, the feature will not be available in the near future and there is no ETA on its availability at this time.

Please stay tuned to our website for further announcements."
Herbert Sitz Send private email
Monday, August 21, 2006
 
 
I totally missed the embedded market.  I'll be moving into that soon (as a hobby - robotics mostly) and I definately need to be able to handle multiple processors and such.
Eric D. Burdo Send private email
Tuesday, August 22, 2006
 
 
It used to be that if an application or processess was too slow you could count on the next generation of CPU to come along and give you a 2X or 3X performance boost. All you had to do was wait for it. Those days appear to be over. Instead of focusing on increasing linear execution speed, chip manufacturers are starting to deliver increases parallel execution speed (i.e. multiple cores). I am certain that we will still see marginal gains in the linear execution speed of new CPUs, but I doubt the Pentium 5 is going to be 2X faster than the P4. Coddled by the exponential growth of linear execution speed, software developers have grown accustomed to their applications getting an automatic performance boost with each new CPU. Those days appear to be over. It looks like we will now need to work for our dramatic performance increases by parallelizing our application. This is not to suggest that every application is going to have to be threaded to the n-th degree. (I mean, there is only so much one can do to a text editor or serial-terminal application to paralellize it.) But I do think that we will start to see a general trend evolve toward parallelism, much like event-driven programming.
A. Nonymous
Tuesday, August 22, 2006
 
 
A.Non -- Great synopsis.
Herbert Sitz Send private email
Tuesday, August 22, 2006
 
 
Herbert Sitz Send private email
Tuesday, August 22, 2006
 
 
Doug Lea is considered an authority in Java concurrency. http://gee.cs.oswego.edu/

Check your favorite university for online course material for a course in distributed programming.

Caltech: http://www.cs.caltech.edu/%7Ecs141/
UC Berkeley: http://www-inst.eecs.berkeley.edu/~cs267/archives.html
genius
Tuesday, August 22, 2006
 
 
For the theoretical background, read "The Igloo Book". Ben-Ari's "Principles of Concurrent Programming" (The first two editions had igloos on the cover).

Why do threaded programming?

Certainly on UNIX systems, one of the fastest ways of knowing IO operations have completed is to do blocking IO -- the kernel will wake the thread up in a fairly timely manner when the IO is done. Blocking IO means doing it one item at a time. Unless you thread the app... in which case you can parallelise the accesses.

Why not use processes? Depends on your level of comms. If the work doesn't need a lot of comms, it'll be fine. If you need a lot of communication, you end up spending a lot of time doing the IPC instead of just sharing the structures.

You can have processes share memory and void the comms, but then you need locking and the interprocess locks are heavyweight compared with interthread locks.

Having said all that, TOH's company typically has two threads; one doing the UI and one doing the work and they just chatter between each other loosely. They do this for UI responsiveness and rarely need to get more complex than that.
Katie Lucas
Wednesday, August 23, 2006
 
 
Two older books I have read:

Multithreading Applications in Win32: The Complete Guide to Threads.
http://www.amazon.com/gp/product/0201442345/

Win32 Multithreaded Programming
http://www.amazon.com/gp/product/1565922964/

The former book was a big hit when it first came out. The old Computer Literacy Bookstore in Sunnyvale, CA (Geek heaven in those days) had stacks and stacks of it and it was "the book" on Win32 multithreading.
MBJ Send private email
Wednesday, August 23, 2006
 
 
Here is the question for the ages:

How do we speed up a programs execution time?
For example: converting a avi file of 700mb to a dvd file of 4.7gb on a p4 3.0ghz with hyper treading 1gb ram pc.Takes the average time of 90 minutes to 120 minutes before it is completed(the burning process is not included).So here we are at the crossroads.How do we make a program or application executions time faster?Is it a question of hardware?Dual core,Quad core or 100 in 1 cpus? Do we need larger L1 L2 L3 caches,bigger front side bus? or is it a question of how we write these programs(concurrent parallel etc)?
I have spent mamy hours of reading.Please clearify this for me .thank you
J Send private email
Tuesday, August 29, 2006
 
 
"How do we speed up a programs execution time?"

There was a time when we could count on intel and amd to come out with faster processors every 18 months.  If something wasn't fast enough we could just wait until the next gen of processors came out and Moore's law would deliver the speed we needed. 

Those days are gone.

Moore's law continues with the number of transistors on the chips doubling but we are not getting substantially more ghz out of the chips.  Instead we are getting multi-core cpus.  This means the only way to take advantage of the new gen of cpus is to write multi-threaded (multi-processor) code. 

So when we have a cpu-bound algorithm like a conversion of a large file from one format to another we will need to break it up into n smaller sub-problems that can be executed in parallel.

Today this can be quite difficult.  The bugs (race conditions, deadlock) introduced by multi-threading don't appear every time and can be difficult to diagnose and correct.

I expect Microsoft and others to produce new languages and frameworks to make this easier.  But it is still worth learning how to do multi-threaded programming now with the basic tools of events, monitors, semaphores etc. for coordinating the work of the threads.
Dan Golick Send private email
Tuesday, September 12, 2006
 
 

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics
 
Powered by FogBugz