The Design of Software (CLOSED)

A public forum for discussing the design of software, from the user interface to the code architecture. Now closed.

The "Design of Software" discussion group has been merged with the main Joel on Software discussion group.

The archives will remain online indefinitely.

Designing Concurrent Software - .NET vs Erlang

So I haven't done any concurrent programming before, but now have an idea for a project that essentially involves spawning a whole lot of 'intelligent' agents that interact with each other concurrently. Erlang immediately presents a very natural way of modeling this application by representing each agent as a process, which then send and receive messages to/from other processes. However, while I am reasonably comfortable with the functional paradigm, I am loath to spend time learning a new platform, especially since I mostly live in Visual Studio. Therefore, I am torn between using Erlang or the .NET way, and since I have limited experience with writing multithreaded applications, I would like to solicit opinions from those who know more than me. If I were to go with .NET, I will likely be using F#, and therefore will attempt to keep my concurrent programming as stateless and lockfree as possible. As far as I can see, the only advantage of using Erlang is the ability to spawn a huge number of processes, but can I just not use the ThreadPool class to optimize the number of threads in .NET?

I will likely have a whole lot of agents (>100,000), so I am really not sure how to go about this.

Thanks!
Tejus Sawjiani Send private email
Saturday, January 06, 2007
 
 
I read about an erlang type library for .net but I couldn't find it. I don't think the functional nature of erlang is really the key, it's more the process model it has and that can be done no problem in .net.

You'll definitely need a cluster for so many agents. My understanding is that the .net threads are very heavy weight (18K) when compared to erlang processes (300-400 bytes).

You could "cheat" by each thread handle a group of state machines rather than using a 1-1 relationship.
son of parnas
Saturday, January 06, 2007
 
 
18K -> 128K
son of parnas
Saturday, January 06, 2007
 
 
We are programming such a highly concurrent server-side application in Erlang although all the other projects use .Net in our company.
And no. ThreadPool is not the answer to your question. It's better than "simple" .net threads in the form that it caches the threads and they don't need to be created every time you need a new thread. But they are as big as "normal" .net thread when they are used.
If you are really ready to use Erlang for this particular project, than go ahead!
foldip Send private email
Saturday, January 06, 2007
 
 
With that many agents (on a single host?) native-thread-per-agent is clearly untenable. I think your Erlang instinct is a good one - you'll need something with lightweight threads. Today that means green/software threads. If Erlang is out, Stackless Python (or PyPy), Termite Scheme, or CL-MUPROC might be worth investigating.

If the MS development toolkit is mandatory (which frankly, sounds like a bad choice to me), you'll need to emulate the actor model of concurrency which can be modeled in non-green-threaded languages (google for a project called Jacob done in Java).
matt.
Saturday, January 06, 2007
 
 
Forgot to mention: implementing agent coordination via a tuplespace might be a useful approach.
matt.
Saturday, January 06, 2007
 
 
Anoher thing you might want to consider is join calculus.  (I haven't tried it.)  E.g.

  http://lambda-the-ultimate.org/node/1931
Reinier Post Send private email
Saturday, January 06, 2007
 
 
http://blogs.msdn.com/oldnewthing/archive/2005/07/29/444912.aspx

"But the real question that is raised whenever somebody asks, 'What's the maximum number of threads that a process can create?' is 'Why are you creating so many threads that this even becomes an issue?'"
Codger
Saturday, January 06, 2007
 
 
> interact with each other concurrently
> I will likely have a whole lot of agents (>100,000)

Whatever you do, don't try to have 100,000 simultaneous threads.

Either two inter-agent messages can happen at *exactly* the same time, or, they can't. If they can't, then you can use even a single hardware thread, and process each message sequentially. If they *can* and *must*, then you'll need to model that fact in your software, because you can't expect PC hardware to model it for you.
Christopher Wells Send private email
Saturday, January 06, 2007
 
 
The problem sounds like a great match for Erlang. If I were you I'd just start coding it in Erlang (pretend you are prototyping). I bet you get so much done you don't even consider .NET.
ronk!
Saturday, January 06, 2007
 
 
Even a massively concurrent bubble sort implmentation doesn't make a bubble sort an intelligent algorithm choice.

Flawed paradigm would be my bet.
Cornfuzzled
Saturday, January 06, 2007
 
 
> 'Why are you creating so many threads that this even becomes an issue?'

The real question is why are threads so pathetically architected that you have to ask? I thread per concurrent activity is a sensible architecture. The only reason people don't do it is the horrible OS support.
son of parnas
Saturday, January 06, 2007
 
 
Thanks for all the answers! Clearly, I have some more research to do here, and look forward to evaluating Stackless Python while also trying to understand if the problem at hand requires messages to be passed between different agents at exactly the same time.

I like Erlang for its built-in elegance in such problems. Thinking about it, the only thing that is preventing me from using it is the lack of an IDE. While I don't particularly have a lot of experience in emacs, I just don't relish the thought of using it for erlang development. That's really why I'm looking for excuses to go with F#/.NET. I guess I'll just have to bite the bullet and attempt to grok emacs.
Tejus Sawjiani Send private email
Sunday, January 07, 2007
 
 
100 000 system threads won't work no matter what you do.

It's not like you really have a choice.
Andrei Send private email
Sunday, January 07, 2007
 
 
Like many people said, don't do 100,000 threads. Ever. I personally am a C++ programmer and have no experience with .NET, but I was told that you can use C++ libraries under .NET - is that true ?

If this is the case, I would definately recommend you to check out the ACE libraries - it implements all kinds of really efficient and scalable concurrent and networked programming concept. In you case, I would be looking into a ThreadPool Reactor design - even if you aren't able to use this library, it's worth taking a look at the design pattern.
Leon Mergen Send private email
Wednesday, January 10, 2007
 
 
You can use C++ libraries from C++/CLI which allows mixing managed and unmanaged code, but you still should keep the two separate as much as possible.  Marshalling data back and forth between managed and unmanaged memory is time-consuming and error-prone.  Personally I'd rather stick with the .NET threading facilities if I were doing .NET programming.
Chris Nahr Send private email
Thursday, January 11, 2007
 
 
Check out the following info about the CCR, which is a part of the Microsoft Robotics Studio.

http://msdn.microsoft.com/msdnmag/issues/06/09/concurrentaffairs/default.aspx

http://channel9.msdn.com/Showpost.aspx?postid=143582

William
William Send private email
Saturday, January 13, 2007
 
 

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics
 
Powered by FogBugz