The Design of Software (CLOSED)

A public forum for discussing the design of software, from the user interface to the code architecture. Now closed.

The "Design of Software" discussion group has been merged with the main Joel on Software discussion group.

The archives will remain online indefinitely.

Design meta-issues

I know that most people will not understand what I am saying. They will think they do and will slam me for not being knowledgable, skilled, elite, or smart enough. But I will say this anyway. Maybe someone will understand.

I have been designing stuff a long time. I am considered by all who know me personally to be among they best they have known.

But I don't know how do design anything really. A problem comes in, and there is nothing to fall back on except half-assed solutions, bitter compromises, or starting from scratch.

This is not the way it is in other disciplines like bridge building. People know how to build bridges. How big is the span, what's the budget, earthquake and wind how much, and do you want it pretty. Get started and two bridge designers might come up with nearly identical solutions. It's more science than art. It's a discipline.

But engineering no. Common wisdom is to use design patterns. Most of these, like factory and singleton, are clumsy patches for language deficiencies. Even the most common design pattern, MVC, you find no two people doing it the same way, and even widespread confusion about what it means, and people implementing things called MVC that are so tighly coupled they might as well be one class. Your only hope is to choose a prebuilt framework in which someone has done it already so you don't have to learn it or understand it. So we have design patterns in Java and in Cocoa and in TK and they are all done differently, even the objects with the same names.

You can choose a nicely designed language that is slow, or a fast language that is a big giant mess and which using simplifying things like its Standard Template Library will drag into your project the most Byzantine architecture imaginable and your design will have to conform to its idiosyncracies and most of your design time will be spent trying to figure out why things aren't working right, or how to conform to its arbitrary peculiarities.

The idea that we can sit down and work at design is only possible within the context of an expert schooled in one particular idiom. The best have their own frameworks that they have written and which no one else fully understands but themselves, if even that. Building all your own custom tools from scratch, different from everyone elses. That's what the experts do.

Anything for more general consumption becomes so overengineered as to require a degree in the framework alone. And that's what schools are doing. Classes in the STL, followed by advanced classes in the STL. And that just gets you a bunch of algorithms and datastructures and does nothing to help with the most difficult part, the interface, or the interaction with remote processes. If you're lucky you go to a vocational school and take vocational classes in beans followed by vocational classes in enterprise beans.

CS programs and research is a big joke. The institutionalized study and invent useless algorithms, thinking it all a branch of abstract conceptual mathematics but with a higher salary. None of the difficult problems of the industry are even looked at but just tossed aside as an exercise for the reader, derided as trivial 'engineering' issues below their lofty consideration. And yet none of these folks could write a program suitable for sale to a customer, and htey have utterly no idea what would be necessary to build something so large as to be made by a team of more than three people. Their best idea is to have one person work on the problem for 30 or 40 years instead because they have no idea how to scale anything.

Case in point. Who can afford to hire the best? Who does hire the best? Microsoft. And doing so not only hasn't helped them design anything, they can't even deliver anything. Not getting down on them here, I'm just saying that if there is one place where we should be seeing these problems solved by intellect or even brute force, it's Redmond. But, following the best known practices has got them in deeper water than those who are just randomly flailing about ignorantly.

It's like the true story about the chimpanzee that consistently picks stocks better than the top stock analysts. Software design is in exactly the same place.
Scott
Saturday, May 27, 2006
 
 
"CS programs and research is a big joke."

I don't know where you went to school, but a large portion of my CS education was dedicated to software engineering.  In the lower-level courses there was a lot of teaching about the *basics*: althorithms, data structures, operating systems, programming languages, and so on.  But beyond that there was a lot of work in software engineering.  One course involved building an entire project using the waterfall method (only 1/5 was coding -- 4/5 documentation and design).  And that was simply one course of many.

There's a lot of work on learning how to manage software projects; about all the possible ways to organize a project.  Courses involving the agile methods as well as the traditional methods were common.  Not to mention learning how to do both high level and low level design.

And there's tonnes of research on this area; in fact, I'd say it's one of the bigger CS research areas mostly because there's so much room for improvement.  Everyone brings up the fact that traditional, bridge building, engineering has been around for thousands of years but software engineering has been around for mere decades.  I still think that's an important point. 

"None of the difficult problems of the industry are even looked at but just tossed aside as an exercise for the reader, derided as trivial 'engineering' issues below their lofty consideration."

That's a very incorrect assumption.  I'm not saying that it doesn't happen, to claim that engineering issues are avoided on masse is simply being ignorant of what is out there.
Almost H. Anonymous Send private email
Saturday, May 27, 2006
 
 
I agree.

After years of trying to "innovate", now I know the prices of things, and I found out that the price is worth it. But...

There's one engineering wisdom that just rings true because it's all that there is to some folks: Copy!

Copy everything that you can, including knowledge, tools, best practices, ideas...

Let's say that software engineering is 80% copying and 20% original content, and the goal would be to have 99% copying and only 1% original content.

That is, engineering should be boring, and when it's not boring, folks complain.

Take Linus Torvalds for example... He says that he is a hobbyist, and that the core Linux hackers are like him, hobbyists.

Generally I agree with what Linus says, I agree with his stances on things... I disagree with him when he says that people should just use KDE... :-) Because GTK is good for my development, and I use Xfce which uses GTK and makes it even greater... But generally I agree with him... For example, when he says that people should use any editor to create software, and not to lose time with the editor itself -- results, results, results...

Yet, the core people working on the Linux kernel are creative folks who rarely can simply "copy things"... i.e., the idea that you can only copy things and be successful is rather wish-thinking... I think managers types prefer these wish-thinking ideals...

Maybe this misconception that software engineering should be boring is what's wrong... For example, if someone like Linus created a software in C++, and he left the job because his boss was a dick and he couldn't stand him anymore, how will the software engineer professional who will assume Linus' responsibilities fair in his role? Not well in most cases... We should have created a garbage recycler to recycle all the garbage that's produced everyday in software development... The problem is, garbage in software development is called softare as well.

Ok, but the garbage could be reduced if people only kept on copying each other, right? Well, if they are copying garbage, it does not help a lot. And even if it's not garbage, you suddenly have all these mountains of multiplied stuff, like a volcano ready to explode.

Design begets design. Quality programming begets quality programming. The contrary is true as well. Like the prophecy that's self-fulfilling, many things can go well because they started well, or they can go wrong because they started wrong. Until we start well most projects, what most people will see is the wrong stuff.

Miniatures! That's what most "enlightened" folks work on. You can't create an Empire State by yourself, but you can create a miniature of the Empire State. The thing is: A miniature is not useful to most people, but maybe you started well, with a prototype for a good Empire State in the future.

Grasp this: Most folks can't create even Miniatures. LOL. :-) hehe. But most folks could copy one, if given enough means.
Lostacular
Saturday, May 27, 2006
 
 
I graduated from Penn State in 1984 with a degree in Comp Sci. I used Fortran 77 on a PDP-11/34 in my first job. In my opinion, things haven't changed much since then.

There have been thousands of press releases for thousands of new products and technologies. Some of these, like 32-bit addressing, have really made a big difference. But most of these "revolutionary" new solutions have fallen by the wayside. We've collectively wasted zillions of hours learning new tools and porting our code. It seems like we're mostly moving sideways.

After 11 years in business as an ISV, I have come to the conclusion that it doesn't pay to use third-party components or libraries. I am currently working to reduce my external dependencies as much as possible. As the original poster seemed to imply, in order to achieve high quality you basically need to build it yourself.

In principle I agree that it makes sense to buy instead of build, and for almost a decade I followed this philosophy. But I have never been completely satisfied with the quality of the third party components I have used, and dealing with the publishers was often very frustrating. In the end, I have decided that I'm better off building many of the components myself.

I recently implemented my own grid control in C++. It was a very satisfying experience. Now I no longer have to worry about my vendor discontinuing the product. And if I don't like the way it looks, I can change it because every pixel is under my direct control. It takes a lot of time to develop a control in C++, but I see it as "pay me now or pay me later".

I realize that most people can't afford to develop their own controls from scratch. But maybe they could if we had better tools to work with. I am very disappointed with the whole .NET thing. A couple of years ago I taught a programming class to high school students. When you look at things from their perspective, you begin to realize just how damned complicated something like Visual Studio really is. I mean, just bring up the MSDN library a search the index for a property name. You'll most likely get scores of references in products that you've never even heard of. Finding the right page of documentation is a daunting task for a newbie. No wonder the number of Computer Science graduates has fallen so precipitously over the past 20 years.
Smoothie
Sunday, May 28, 2006
 
 
>> After 11 years in business as an ISV, I have come to the conclusion that it doesn't pay to use third-party components or libraries.

I've been coming to the same conclusion, all the difficult problems I have had to deal with are related to third party libraries or interfacing to external applications.
Tony
Sunday, May 28, 2006
 
 
Writing software is an act of creation and creation can't be made into a cook book. You sound dissapointed that you couldn't be replaced by a robot. Why would you want a job where you could just follow a recipe?

Isn't the fun thing about programming is that you have to make yourself better to be better?
son of parnas
Sunday, May 28, 2006
 
 
Wow, Scott, what an arrogant post. First you assume that most of us won't know what you're talking about because you're so much better than us, and then you go on to apply your obviously limited experience to tarnish the entire software industry. Way to go.

Oh, and you tried to curry public opinion by bashing Microsoft too. Shame you didn't use Google as a case in point, that would have blown your argument out of the water. But I guess supporting your argument wasn't the point, was it?

But to get on topic, this isn't the sort of thing professionals worry about, we just get on with the job. We're too busy working to be digging fluff out of our navels.
I says it like it is.
Sunday, May 28, 2006
 
 
> Most of these, like factory and singleton, are clumsy patches for language deficiencies.

I call clueless amateur.

For all the flaws with specific design patterns, all a "design pattern" actually is is a description of a problem, a description of a solution, and a list of tradeoffs involved in that solution. Oh, yes, it also comes with a name, so that people can actually communicate quickly and not need to spend a half day meeting re-inventing a solution to a previously solved problem.

Moving the solution from a library to a language feature is completely and totally irrelevant. Finding a better solution may be a great idea, but it's funny that for all the design pattern bitch-fests, there's millions of people who "know" that the patterns are awful, but can't actually give a better solution for the problem.

Hell, anyone who's actually read the GoF book will be aware that most of the patterns described have a comment mentioning a language in which the particular idea is a language feature, languages that include it in standard libraries, and which languages make it particularly awkward.


> You can choose a nicely designed language that is slow, or a fast language that is a big giant mess and which using simplifying things like its Standard Template Library will drag into your project the most Byzantine architecture imaginable

I call "idiot".

The "Standard Template Library" is obviously a C++ whine (I don't know any other language that uses that particular name for a library), and it's hard to think of a modern language (eg Java, C#, python) that hasn't been called "nicely designed" that doesn't have a library with (sometimes) many thousands of classes. To grizzle that the STL imposes structure while assuming that the object librareis in other languages have no structure at all is idiotic, at best.

We can spend the next thousand years whining and bitching like children about which one is best, but any competent and intelligent person will realise that use of a library (be it templates, a library of objects, a library of C functions, whatever) will involve tradeoffs, and that inevitably there will be a conflict between the structure of the library and the person preferenecs of the user.

Only the retarded offspring of a village idiot and a TV weather girl would be unaware that this applies to every programming language and library that has ever and will ever be devised.

Sunday, May 28, 2006
 
 
>>But to get on topic, this isn't the sort of thing professionals worry about, we just get on with the job.

I strongly disagree. Writing software with today's programming tools is like building a car using a steam engine. At some point in the (hopefully near) future, an alternative approach to programming will be devised that will replace today's tedious and time-wasting tools. The future won't be built by people who accept today's limitations as a fact of life.
Smoothie
Tuesday, May 30, 2006
 
 
The point is what you are trying to mimick.
A bridge mimicks something that has a single function, stay on top of an empty space. This may be achieved by a fallen tree as well.
The engineering metaphor breaks if what is mimicked is human behaviour or the human way of organizing information.
This is WAY more complex.
An ERP is simply something that's striving to mimick what humans do in their job before the ERP. There is nothing done qualitatively different because of informatization. It's incredible how many objects and process, and how complicated, a human brain can handle implicitly.
Sevenoaks Send private email
Tuesday, May 30, 2006
 
 
Clearly the OP has not seen engineering as practiced by the Army Corps of Engineers.

I don't see the point of arguing about why we cannot achieve perfection.

There are always compromises, and there are never going to be any processes guaranteed to produce success.  It's a job done by humans because machines can't do it, and humans have some major problems.
Cade Roux Send private email
Tuesday, May 30, 2006
 
 
>>The engineering metaphor breaks if what is mimicked is human behaviour or the human way of organizing information.
This is WAY more complex.

Agreed. I believe that good software is like a good employee. What you really want to do is to delegate a task to the employee. That's why "works independently" shows up on most performance evaluation forms. If an employee constantly comes back to ask you for input and can't make any decisions for himself/herself, then they aren't adding much value. Good software takes responsibility for the task that we delegate to it.

Case it point:  When copying a folder from drive C to drive D on a Windows system, what happens when it hits a file that is read-only? It throws up a message box asking if you really want to copy that file. Suppose you're out of the office and there are 5,000 additional files not yet copied. Does it keep copying the rest of the files while it awaits your input on the special files? Nope. It just sits there and twiddles its thumbs. So when you return to the office, the task that you delegated has barely even been started. If an employee behaved in that way, we wouldn't put up with it. So why do we put up with software that behaves so poorly? Probably because at some level we realize that writing software is a really complex job.

How complex is it really? One measure would be the number of symbols needed to represent the design. Let's say we had the complete design specifications for the Golden Gate Bridge in one folder, and all of source code for a commercial accounting package in another folder. What's the relative size of those folders?

I'm not a civil engineer. Perhaps the bridge design specs would be a lot bigger than I am imagining.

Information theory tells us that any given message contains a specific quantity of information, and we can express that quantity as a specific number of binary bits. Engineering design specs and software source code are just two types of messages. Good compression algorithms have the goal of storing a given message in the minimum possible number of bits.

So here's an experiment:

Compress the source code for a large software project using every available lossless compression algorithm. Find the minimum file size. Now do the same thing with the design specs for a large engineering project. Compare the file sizes. This should provide an objective measure of the relative information content of each project.

Einstein's famous message E = mc2 requires only five symbols. However, it blew away a whole lot of complex equations that previously represented the best available explanation of how the universe behaved. What Einstein indirectly accomplished was to provide a concise way for people to express truths about the universe.

What we need is an Einstein who will provide us with a better programming paradigm that provides the same sort of conciseness.

Suppose we took an existing software application developed using pre-.NET tools. Then we rewrite the same application using C#, following all of the best practices. When we're completely finished, we compress the source code for the old and new projects and compare their compressed sizes (ni order to approximately compare their information content). Would the .NET project be smaller than the pre-.NET project? How much smaller? If it's not significantly smaller, then we're not really moving forward, are we?

I realize that there is going to be a certain minimum size dicvtated by the inherent complexity of the problem being solved by the software. But my gut feeling is that current software methodologies require us to spend 10x as much time and write 10x as much code as an optimal system would require.
Smoothie
Tuesday, May 30, 2006
 
 
"People know how to build bridges ...
But engineering no. Common wisdom is to use design patterns ... clumsy patches for language deficiencies."

Oh? Kile how "arch" and "suspension" patterns are clumsy patches for the finite strength of iron?
Java GUI Programmer
Wednesday, May 31, 2006
 
 
"Kile" was meant to be "Like". (I get race conditions when typing with two hands!)
Java GUI Programmer
Wednesday, May 31, 2006
 
 
"The future won't be built by people who accept today's limitations as a fact of life."

Oh Smoothie, you are so deluded. Get used to the phrase "Do you want fries with that?" - it will be a big part of your professional life.

Let me guess, you're either a student or English is not your first language. Either way, you haven't worked in the real business world, have you? It _really_ shows.
I says it like it is.
Wednesday, May 31, 2006
 
 
>>"The future won't be built by people who accept today's limitations as a fact of life."

>>Oh Smoothie, you are so deluded. Get used to the phrase "Do you want fries with that?" - it will be a big part of your professional life.

>>Let me guess, you're either a student or English is not your first language. Either way, you haven't worked in the real business world, have you? It _really_ shows.


Hah! That's pretty funny! FYI, I was born and raised in the USA, got by B.S. in Computer Science from Penn State in 1984, wrote software for a research lab for 5 years, developed databases for a 50-person software company for 5 more years, and have operated a Micro ISV for the past 11 years. I am 46 now and I figure I'm about halfway through my working years. Apparently my youthful optimism makes me seem a lot younger. I'll take that as a compliment!
Smoothie
Thursday, June 01, 2006
 
 
Glad you find it funny, Smoothie. Me, I kind of pity you.

Still, I take my hat off to you for surviving so long in the business with that attitude. I wonder how much more successful you would have been without it.
I says it like it is.
Thursday, June 01, 2006
 
 
I guess that depends on your definition of success.
Smoothie
Thursday, June 01, 2006
 
 

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics
 
Powered by FogBugz