The Joel on Software Discussion Group (CLOSED)

A place to discuss Joel on Software. Now closed.

This community works best when people use their real names. Please register for a free account.

Other Groups:
Joel on Software
Business of Software
Design of Software (CLOSED)
.NET Questions (CLOSED)
TechInterview.org
CityDesk
FogBugz
Fog Creek Copilot


The Old Forum


Your hosts:
Albert D. Kallal
Li-Fan Chen
Stephen Jones

What are the 2 biggest problems of Software?

Many laymen think cure to cancer and cure to AIDS are two biggest outstanding problems for the researchers in medicine.

    Likewise, what are the two biggest outstanding problems for the researchers in software?

Let me name my list:
1.    True artificial intelligence. Most artificial intelligence researchers agree that, today artificial intelligence is nothing but few complex algorithms. Hence they admit True artificial intelligence not yet invented.
2.    Software engineering based on true components and component oriented programming, which could increase the productivity by 10 folds and make software development more predictable, accurately measure productivity or estimate costs just like any other true engineering disciplines such as mechanical, hardware or civil engineering. Many agree that software engineering is not true engineering. For example, US government try to crack this problem nearly 15 years back but failed: http://www.atp.nist.gov/focus/cbs.htm
 
I like to know from others, what they feel are the two biggest outstanding problems for the researchers in software?
Satya Send private email
Wednesday, May 21, 2008
 
 
Both of your suggestions are closely related, and pipe dreams, as they are essentially rooted in the understandable but still odd notion that computers should be able to do what we do.

The biggest most costly issues in software development are lack of compatibility and the integration woes of compatible components (e.g. a bug in a 3rd party binary component). But I am not saying the situation will be solved or even improved in the next 100 years. This is a good thing, because super-efficiency in computing would be dangerous.
Ben Bryant Send private email
Wednesday, May 21, 2008
 
 
> "pipe dreams"

Maybe so, but both of those are still an excellent way to get a research grant.

The secret is to always be on the cusp of a breakthrough.
Jimmy Jones
Wednesday, May 21, 2008
 
 
Two biggest problems of software:

1) Programmers
2) Users

Sounds a little flippant, but as long as those 2 "problems" exist software will be less than perfect.
The Original Henry
Wednesday, May 21, 2008
 
 
@Henry - you beat me on the Programmers part but it's really "Crappy Programmers" - and I don't think users are the problem - at least users aren't quite the problem that well designed software cannot deal with!

AI fundamentals were bogus to begin with and people only realized that later. Something that more closely resembles the workings of humans someday might come a bit more closer to human behavior and thinking. (http://numenta.com - Jeff Hawkins covers this all in great details - On Intelligence.)

Adding to the list - I think programming languages are still very "machine oriented" instead of "human oriented" w.r.t expression and precision required. You can't still express a human mind effectively using a programming language unless you also happen to have the "machine mind" as your second nature. This is a significant problem with mass producing quality software. This is part of the reason many people can't seem to get programming right - they lack the machine mindset.
annu_gogte
Wednesday, May 21, 2008
 
 
Two biggest issue that are in constant conflict aqnd keep all software cycling back and forth is a need for ever greater simplicity and yet greater and greater flexibility.

More and more people want and therefore designer simpler and more intuitive products that make our lives easier and more efficient, including the work developes do. So I see a large need for simpler more condensed software. Thats inline with components, etc. At the other end is a growing need for more custom development, features, and thus software thats more flexible. Visual Studio, as a software tool, or Word allow more and more people to do more and more, but more features and buttons and object-rich languages add more complexity and ultimately large, convoluted frameworks and enterprise systems that eventually have large volumes of documentation and training to support.

Simplicity = 1/Flexibility

As complexity increases with flexible parts, so simplicity goes out the door. Thats the central problem with all software, I think. Most really good architects and software engineers know, like everything else in this world, there is a "sweet spot" to all projects, and strive for a healthy balance in that equation, I think. Allot of the very left or right of center architects Ive worked with go one way or the other and run into problems eventually. I worked with two people....one was pro custom library, the other open source plug and pray libraries downloaded from Microsoft and the web. In the end, I was with the custom library person and that was a good direction for us. When I look back though, I do see how our more flexible, customizeable architecture might have benefited from a simpler standardized architecture. So, trade offs all around.
Ranger
Wednesday, May 21, 2008
 
 
"1) Programmers"
"2) Users"

That was my first reaction too.... :-)

Just need to add "Microsoft" to the trifecta and it will be complete.
Jimmy Jones
Wednesday, May 21, 2008
 
 
1) the move away from transparency
2) the big moves from technology to technology being driven by commercial interests.

For the 1st:
I've found that 90% of my time in the MS world is figuring out how to convince MS's technology to do what I want.  Which buttons do I push and in what order... I'm exaggerating, but not by enough for comfort.

For the 2nd:
Technology is no longer tested by what works best, it's tested by which company has the best marketing team.  This isn't completely 100% true, but as above, it isn't far off the mark.
Fake Programmer
Wednesday, May 21, 2008
 
 
I agree Fake. Seems like we have moved wholesale to an undervaluation of IT, to the point where we have glofclub swinging salesmen selling stuff they have never even used themselves. And we have CEO's and marketers pushing stuff before its tested and ready. People dont care about quality and as a result, they end up paying twice as much years later managing and hiring more people to fix software. If we would stoff offshoring so much stuff and hire talent and innovators in IT and carefully work solutions around high quality innovative engineering, we would see not only higher profits and adoption in corps, but higher quality which reduces overall cost over the years a software app is supported in the open market.

People have forgotten how to measure ROI correctly in software. I guess they assume because IT people are a commodity that can be bought and sold to the cheapest bidder, they undercut IT as a whole, including software.
Ranger
Thursday, May 22, 2008
 
 
From a development staff perspective, the biggest problem I see is that the management does not understand the technology challenges well enough.  This leads to huge amounts of inefficiencies, redundancies, solving the same problems again and again, not solving some problems ever, group think, make-work.  Looking busy is more important than actually doing something useful.  Fire and motion?
Milo
Thursday, May 22, 2008
 
 
1. This is a relatively young discipline (compared to the history of mankind :) and it has yet to widely embrace the quality standards which have been established by a minority.

2. It is also a business and many people either:
- are reluctant to treat it as such; or
- do treat it as such but are nonetheless unable to manage this or any other business in the first place

Just my two cents. I'm not sure that I'd cut the list at two or, if made longer, that I'd keep these two at the top.
Daniel_DL Send private email
Thursday, May 22, 2008
 
 
I'm not clear as to what "true components" would be - every time I develop anything I am re-using operating system components, hardware components, libraries and services accessed over a network.

How would a "true component" differ in specific meaninful ways from these existing components?

As for emulating human level general intelligence I personally think (and I worked in AI research for six years) that we haven't even begun to define the problem well enough - let alone design a solution. Unless we do this by building a high-fidelity neurobiological simulation engine and scanning the "running" state of a brain into it I suspect we are at least a century away from constructing a general artifical intelligence from first principles.
Arethuza
Thursday, May 22, 2008
 
 
Bryant,

    I respectfully disagree with you. I think the second problem certainly will be solved in less than 10 years. New compilers, dynamic linkers and/or disassembles for loosely coupled true components may require for solving this.

However the first problem AI is very hard to crack, but might be solved in about 50 years. This requires completely new train of thought, a seed for new thinking process, which takes in completely different path.

When nature can do that, why can’t we do that? I feel, it requires new type of chips that emulate nature’s building blocks such as neurons or something. Then build new type of data repositories and models to process data/algorithms.

This seed for true AI might happen by accident or serendipitously. Of course, then it will be a long and arduous process may take decades to grow from the seeds to evolve the intelligent (e.g. such as natural instincts) level of a dog or a bird.
Satya Send private email
Thursday, May 22, 2008
 
 
Arethuza,

    I agree with you on the AI. There is lot of work need to be done, starting from scratch. I feel, we haven’t even found the starting point/path, which could leads to the AI. I feel, often such starting points discovered by accident by non-software expert (may be junior with limited expertise, who miss-understand a theory, which leads him in a new undiscovered path).

    As per “True Components”, there are many people have different views of how true components would be (none of the views accepted yet by research community). But one interesting proposal is found at wikipedia page for software components: http://www.cbsdf.com/ps_blog/SE-MissingLink.htm and
http://www.cbsdf.com/ps_blog/Component-characteristics.htm

    The above pages would make sense in light of the following sample application build by assembling such Replaceable Components: http://www.cbsdf.com/ps_blog/WhyRepCom.htm.

Caution: Although, I feel theoretical it is possible, I am not sure such components and component assembling is possible today.
Satya Send private email
Thursday, May 22, 2008
 
 
As I see it, I wouldn't focus on what constitutes problems NOW being experienced in software engineering because the issues of today will be the solutions of tomorrow.

I'd rather focus on future innovations and think about what IS possible for the immediate future.

As it relates to AI, I think what you will see (and I do recognize that today it cannot be sustained with the current state of technology), is an emergence of new systems that can actually accommodate AI methodologies and implementations by way of biologically based programmatic systems.

This will be achieved when computer science converges with genetic engineering. It's already started happening and within the next decade, (I predict) there will be innovative tools used to actually program biological systems.

Once this occurs, the problems outlined in this thread will be problems no more. However, what will replace them are even more complex and harrowing challenges that need to be addressed...
Brice Richard Send private email
Thursday, May 22, 2008
 
 
In the space of business software, I don't ever see a global set of 'true components.'  Every company does work differently, heck, look at major ERP installations, no one says 'sure, we'll just run it stock out of the box' everything is customized to the way that particular business works.

Now, with companies installing ERP systems and shutting down multiple legacy systems, we are moving to a point where, for a particular company, it's likely that they will have one data model, allowing for a company-specific 'component' that operates on a data element to start to be reused within the company.
Tim Send private email
Thursday, May 22, 2008
 
 
I don't know why I'm replying to such a silly question...

The state of AI is no closer to a "thinking machine" but I think we forget how much things have progressed. If you brought someone from 1960 to view our technology and how much it adapts itself to our requirements dynamically, they would probably point and say "aha, artificial intelligence".

As for components, I think this is an incorrect goal. Just look at components for automobiles. By using top down design, an automaker can design a component (gas pump, say) that is interchangeable in a line of automobiles for a relatively short span of time (maybe 5 years). What is the state space of a gas pump (how many different regimes does it have to deal with in the various applications in which it is used)? A small number. Maybe 20 or something. Any non-trivial software application is 2-3 orders of magnitude more complex. I think it's highly unlikely that widely reusable components are possible without a thinking machine to configure them. In which case, why am I even involved? Let the machine write the code and I will sit on a beach and drink rum.
Financial Programmer
Thursday, May 22, 2008
 
 
The two biggest problems in software are:

1. The barrier to entry is too low
2. Code is harder to read than to write

The first problem is counterintuitive. Isn't it great that you can invent the next big thing in your garage? But the problem is that the entire history of building complicated things before software required a huge investment of time and money. This high cost forced people to be especially careful since there was so much riding on success. Software is complicated machinery that can be made cheaply in a world where all other complicated machinery cannot.

The second problem is somewhat related to the first. Even if we acknowledge that there is lots of bad code out there, it's easier to write more code than it is to understand what exists. As a result, we're likely to try and rebuild or reinvent rather than improve, modify or refactor. This is unlike all other building-complicated-things disciplines, where people almost always use components and draw upon past designs.

I don't know how we're going to solve these problems. If anything, they are getting worse, not better.
Robby Slaughter Send private email
Thursday, May 22, 2008
 
 
AI is not a "software problem" - we should not be distracted from the theoretical nature of AI by its popular medium of implementation, software. A true understanding of intelligence would easily be a separate field of its own. A similar example I'd say is control theory, which exists as a body of knowledge distinct from the physical realization of control systems.

That said, I think that the biggest problem in "software" proper is that 90+% of the field's practitioners would be considered quacks when standards of professional excellence emerge say 50+ years from now. In medicine, for instance, I've heard that there used to be a time when in certain regions barbers made themselves useful as surgeons because they were "good with blades and scissors". Just imagine - there was a time when the idea of infection was unknown in medicine! Armies of nations were a very amateurish affair once upon a time - if you were an able bodied male, you essentially were an instant soldier.
Arun
Thursday, May 22, 2008
 
 
> Code is harder to read than to write

+1 to this, but this is also mostly because source-code comprehension is simply not explicitly recognized and cultivated as a programming skill. There is too much unhealthy emphasis on writing readable code, but very little focus on *reading* existing code.

Ditto for debugging.
Arun
Thursday, May 22, 2008
 
 
The two 2 biggest problems are:

1) crappy programming languages
2) crappy operating systems based on the crappy programming languages
Achilleas Margaritis Send private email
Thursday, May 22, 2008
 
 
> Code is harder to read than to write

+1 for this

> This is unlike all other building-complicated-things disciplines, where people almost always use components and draw upon past designs.

It is very good and interesting point. True software components must accommodate this!
Satya
Thursday, May 22, 2008
 
 
No two people think the same.

Read The Mythical Man Month.

Depending on who you ask All of this occured at the Tower of Babel (The Bible).

http://en.wikipedia.org/wiki/Tower_of_Babel

http://discuss.joelonsoftware.com/default.asp?joel.3.628294.11

What advances could be made if code were as easily readable as it is writable?

What advances could be made if all code were one language?
Scruff McGRuff Send private email
Thursday, May 22, 2008
 
 
"Simplicity = 1/Flexibility"

That's awesome. I'm stealing that.
Drew Kime Send private email
Thursday, May 22, 2008
 
 
1. Programmers who use VB.
2. Programmers who use VB and only know variants.
Johnny Moondog
Thursday, May 22, 2008
 
 
And yet VB solved the component problem by making use of at least a subset of COM/DCOM easy and natural.
So tired
Thursday, May 22, 2008
 
 
1. Self-deception perhaps? (It applies to all fields.)

This leads people (especially non-technical managers) to imagine that speed, cost and flexibility are not tradeoffs but rather all them can be had without affecting the other. Well its called a tradeoff triangle for a reason. Guess they don't teach that in business school. Self deception has people astray to search the ends of the world for the Holy Grail and warewolf hunter the Silver Bullet. Not realising the problem is themselves thus they will never find they cure because they themselves are the problem.

Imagine you are a warewolf, but after each time you go on rampage and kill innocent people, you completly forget about it. In fact, when you are back being human, you are the one investigating the warewolf killings in your town. Every clue will lead you to yourself being the culprit, but can your accept the reality or would your deceive youself? That's self-deception folks. It turns well meaning folk into monsters such as Dilbert's pointy haired boss.

Many IT managers are seduced by the idea of The Ultimate Cure or the Final Solution.(And who could blame them?)
But what is the desease? Nobody seems to know. Hmmmmm.

;-)

2. Sacrificing the long-term for the short-term. The world is not going to end if your software doesn't ship in time. (alright, you might lose your job, but you do have a choice to leave, or start your own company)

However compromising quality is a far worse sin than missing the deadline. Because a screwed up piece of software is like a engineered virus gone wrong; it stays and torment its victims long after its creators are long gone: long term collateral damage. It causes morale necrosis and ultimately leads to management dementia.
Gene Quah
Thursday, May 22, 2008
 
 
The Original Henry:
"
1) Programmers
2) Users
"

Although funny, this is actually not far from the truth.

I think that the biggest problem is the "impedance mis-match" between what Users want/need and what Programmers deliver.

The second biggest problem is the time lag between starting a project and getting it in Production.

Of course, there have been numerous developments in both of these areas over the decades, but I get the feeling that these are baby steps and that we aren't even close yet.

It seems to me that a giant leap in the right direction would be to take programmers out of the equation as much as possible, and just allow the users to build their own software.

I know this is seen as heresy by many, but I believe that it can, and does work. It also neatly side-steps the two problems identified above.

I have created a system that allows the users to create their own business applications. Screens, tasks, data-sources, reports, etc, are created by the users in a declarative way and then executed by a tiny runtime. In a recent implementation, a non-trivial system was created with over a thousand screens in just a few months, by two users.

I challenge any developer to be that productive. A 1,000 screen business application, designed, implemented and in production, from scratch, in six months. I'm not talking about trivial CRUD screens here, but complex workflows, capable of doing non-trivial work. I know I couldn't do it in the traditional way.

The point is, in common with most other fields, you have to work smarter, not just faster.
Odysseus Send private email
Friday, May 23, 2008
 
 
Quote:

"I challenge any developer to be that productive. A 1,000 screen business application, designed, implemented and in production, from scratch, in six months. I'm not talking about trivial CRUD screens here, but complex workflows, capable of doing non-trivial work. I know I couldn't do it in the traditional way."

I am involved in the development of a business application project that has taken 6 months to develop around 20 screens of complex workflow functionality.
Achilleas Margaritis Send private email
Friday, May 23, 2008
 
 
1. Complexity
2. Concurrency
MoffDub Send private email
Saturday, May 24, 2008
 
 
Just one problem:

"assumptions"

or maybe two:

"humans aren't logical, computers are"
Marck Send private email
Saturday, May 24, 2008
 
 
Odysseus said:

I have created a system that allows the users to create their own business applications. Screens, tasks, data-sources, reports, etc, are created by the users in a declarative way and then executed by a tiny runtime. In a recent implementation, a non-trivial system was created with over a thousand screens in just a few months, by two users.

I challenge any developer to be that productive. A 1,000 screen business application, designed, implemented and in production, from scratch, in six months. I'm not talking about trivial CRUD screens here, but complex workflows, capable of doing non-trivial work. I know I couldn't do it in the traditional way.

My question:

I don’t understand what this mean? This requires one person to create complex screens an hour (assuming 8 hours a day and 250 working days a year). No single human being could possible design or even comprehend so many workflow use paths in a year.

I feel, it is best to back this kind of “outrageous” claims by what kind of break through inventions you made (full details including minute points). Some of us don’t mind spending days or weeks to understand such break thorough inventions/principles.

I feel, no tool based on existing knowledge base could accomplish that. Whatever tool that can accomplish such task must be based on new disruptive invention/discovery, which likely expose fundamental flaws in the software engineering and/or prove widely accepted principles such as Dr. Brooks ‘no-silver-bullet’ or ‘mythical-man-month’.

For example, Satya posted couple of web pages (e.g. http://www.cbsdf.com/ps_blog/SE-MissingLink.htm) from a website that only claims 2 to 4 times productivity gains in building just RIA only, which claims to discovered True components.

It would be helpful to others, if you could explain how your tool could make developers over 100 to 200 times productive (Achilleas said above: It took six months to create 20 screens and in software complexity grows exponentially with size and number of developers involved).
John Send private email
Saturday, May 24, 2008
 
 
@John:
It is what it is. I simply describe it. You may call it "outrageous", or not, at you own whim.

You are partially correct, in that it is based upon an invention, of sorts. Although, when you say "no tool based on existing knowledge base could accomplish that", of who's knowledge do you speak?

Additionally, when you say "No single human being could possible design or even comprehend so many workflow use paths in a year" are you sure you speak for the whole human race, or is it a smaller constituency?

I never claimed that my system made anyone 100-200 times more productive. You came to that (incorrect) conclusion on your own. If you read what I wrote, then what Achilleas Margaritis wrote, this simply does not follow.

I stated that the screens involved in my User's system were not simply CRUD screens. Obviously, any decent developer could create a 1,000 screen CRUD application in a day, if they had 250 tables and wanted screens for create, read, update and delete.

I meant that it was more than that, and included complex workflows, business rules, etc.

I suspect that the work of Achilleas Margaritis was far more complex, as you would expect on the spectrum of possible projects and their solutions. So, I do not feel that your analysis holds.

You close with the comment "software complexity grows exponentially with size and number of developers involved", which is not true in all cases, although it certainly can be. However, you neglect to realise from my descriptions that the "number of developers involved" is zero.

This is the whole point of my original post about what The Original Henry had said. It is also the heart and soul of the system that I have created. It takes developers out of the equation and so removes many of the usual problems of communication and complexity explosion noted so eloquently in the seminal work that you refer too.

As I have stated above, I know I speak heresy here, and fully expect the "appropriate" response. Conveniently, this does not change the fact that the system exists.
Odysseus Send private email
Monday, May 26, 2008
 
 
I opened this thread and suddenly felt a cool breeze coming from my monitor. Curious as to the source of this phenomenon, I read through to the end, and there I found the answer. It was the furious handwaving.
Drew Kime Send private email
Tuesday, May 27, 2008
 
 

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics
 
Powered by FogBugz