The Design of Software (CLOSED)

A public forum for discussing the design of software, from the user interface to the code architecture. Now closed.

The "Design of Software" discussion group has been merged with the main Joel on Software discussion group.

The archives will remain online indefinitely.

The application/database divide

Being a totally database-oriented person (ten years of Oracle) I read much on the subject of the problems of letting java (etc) developers touch databases, and on the problems that often seem to plague attempts to create "database agnostic" products (concurrency, scalability etc).

Would anyone like to share a peek at the other side of the coin? For example, I've heard it said that database developers appear to be allergic to source control, and that the primary job of DBA's is to say "no".

Thoughts?

Dave
David Aldridge Send private email
Tuesday, December 28, 2004
 
 
I assume the other side of the coin you're looking for is a developer's perspective?

Here you go:

In my experience (15 years of software development with and w/o databases in C, C++, and Java primarily), as it pertains to working with database administrators/developers:

- Many DBAs don't understand the concept of business value and believe that applications are blood-sucking leeches attached to the pure database.  Not that data isn't valuable, but its often useless without the applications that feed and anaylze it.  This seem obvious, but I've worked with too many folks who didn't understand this.

- An inexpereinced Java, C#, etc. developer working with a database can fall victim to the "leaky abstraction" problem where they treat the database like a black box with infinite performance and scalability.  I've never met an experienced developer who did this.  Certainly vendors have contributed to the problem by pushing abstract specs (EJB entity beans for example) that can be poorly implemented and thus produce poor performing apps when used blindly.

- I actually think a good DBA is very valuable to a software development team, but a bad DBA adds painful overhead to the process, adding little value. SQL Server DBAs have been consistently bad in my expereince (doesn't mean that's a rule, just a disturbing trend).

- I have worked with several DBAs who didn't understand versioning, source control, etc., and thought just having 2 or 3 database instances was sufficient. In my opinion, any DAB who doesn't believe in writing scripts for database creations and migrations is living dangerously.
Dave C Send private email
Tuesday, December 28, 2004
 
 
I personally think that a RDBMS is simply a mechanism, one of many kinds, that is used to store information.  I say one of many kinds because if you abstract the idea into a multi-tier problem, storage, business, and front-end, you can easily replace the storage portion with anything other than a RDBMS, files for example.  Granted the DBs are generally more sufficient and optimized for large quantities of data, but it could still be done other ways. If you're using the DB for more than just storage by using stored procs, triggers, or hooking other DB dependent systems onto it, then you really do have to pay attention to the schema.  But, if you're putting all of your business and reporting logic into objects, like they should be, then you, the developer, are free to do whatever with the back-end. I mean, who cares? 
  Having worked on both types of systems, business logic in procs and relationships in foreign keys versus all business logic and relationships maintained in java objects, I can definitely say that the more functionally scalable and malleable system is the latter.
Bretttt Send private email
Tuesday, December 28, 2004
 
 
DBA's get the "is this in fifth Normal Form?" question, which sucks.  DBA's get to do in Stored Procedures what a procedural programmer could easily do in a Procedure -- but the result is much less data going over the network and much better application performance.

The Database is going to be a central bottle-neck of any application.  The question is, who has the power to change the bottle-neck.  Programmers in theory don't know what it takes to optimize query's, or optimize the database to better support the common query's they're going to have.  DBA's do (in theory).

We're back to leaky abstractions, actually.  You don't want a Java programmer treating database records as 'just another object', because there are performance implications if you do.  If you treat a database table column value as 'just another variable', the Java program is going to thrash your database.

As long as the Java programmer respects appropriate use of the Database (it's a Data Repository.  It does Queries really well.  Inserts Slow, Queries Fast.  It has permanence.  It has NO History.) then things should work out.  As long as the DBA doesn't become power-mad (NO! NO! It's MY Table Structure!  I WON'T change it!  You Can't Make Me!  You'll Break It and I'll Get Blamed!) again things should work out.
AllanL5
Tuesday, December 28, 2004
 
 
Thanks guys -- good comments.

Here's a link to a very Oracle-centric view, just fyi ... http://asktom.oracle.com/pls/ask/f?p=4950:8:::::F4950_P8_DISPLAYID:883929178230

I wonder whether some of these issues just come down to the same old problem -- hired the wrong person. The obstrucive DBA, the PL/SQL developer who doesn't believe in source control, the java developer who doesn't know the difference between "select count(*) ..." and "select count(my_column) ..." etc..

Now once you get beyond that issue (and I'd admit that to be rather a theoretical exercise in most companies), I'm interested in what technical factors you see that would lead you to avoid a database-centric architecture? As Tom Kyte points out in the link above, recent history tends to suggest that it is the front-end technology that is more likely to vary than the database. If you wrote an application fifteen years ago in PL/SQL, then you would have updated it to take advantage of new optimizations and made changes according to business evolution, but the technology would be broadly the same except for the front-end. Now, if you'd built a thick client in VB or (insert name of fashionable interface technology from past ten years) you would almost certainly have changed it completely since then at least once, no?
David Aldridge Send private email
Tuesday, December 28, 2004
 
 
Oh, let me add that I've worked with DBA's myself who couldn't actually write a SQL insert statement -- shocking, eh?
David Aldridge Send private email
Tuesday, December 28, 2004
 
 
Brett, I have to say (and I don't mean this to be a flame or to be unpleasant etc) that it sounds like you haven't worked on any kind of large database project -- is that so? I mean, how do you handle concurrency, locking, recovery, indexing etc with a series of flat files for data storage? Do you generally write your own code to maintain primary/foreign/unique keys etc?
David Aldridge Send private email
Tuesday, December 28, 2004
 
 
I do have to say that my current project has a limited number of users, 6 at most, but a large DB of 150 data tables, not lookup tables.  So DB contingency is not much of an issue.  A smart design on the domain objects and a framework for persisting can be just as effective for limiting these issues, too.  I have a persistence layer which is based on a strategy pattern, so each domain object can have its own storage type if necessary.  The objects stored in the DB generally have a TOPLink project built for them, but some are retrieved by straight JDBC.  Configuration files and such, are retrieved and cached using a file strategy.  Some other systems in the business are accessed via HTTP:get and have that type of a retrieval strategy.  Anyway, you do have a point about indexing, but again post retrieval caching strategies can be used to minimize the amount of hits on the DB.  I haven't had any contingency issues yet.  DB retrieval issues have been solved with indexes.

The largest 2 tier project I've worked on had a larger number of users, 20, but only around 70 tables which were mostly lookups.  This worked great for the designed problem but extending it to new functionality just plain sucked.  I had to create SQL unit testing framework in order to move ahead. But development in that environment was no fun, too much stress.

Did that answer the "How?"
Bretttt Send private email
Tuesday, December 28, 2004
 
 
Yes, Brett, I think those were all 'small' database applications.  I've worked on an IRS call-center quality metric generation project.  We has 28 million records per day, which had to be ingested into 28 different metric tables.  I've worked on NASA data collection sites, with 1 Terabyte/Month of data gathered and qualified.

'Size' is relative here, also.  You can have your 150 tables (which is large, I would guess) but you also need the number of records inserted per day, along with the number of concurrent users (websites can get thousands) and number of transactions.

Don't belittle the role of databases by saying they can be replaced by flat-file implementations.  Logically they can, and for all of your apps, perhaps, but there's a big world out there.
AllanL5
Tuesday, December 28, 2004
 
 
I apologize if I came across as belittling DBAs.  There is a definite need for techies who know how to drive those tools.  It's become a very mature product, RDBMS, but remember that the whole of this product is built on software.  So to think that it can't be replaced by another, completely innovative technology is somewhat short sighted. 
It's not that it should be replaced or that it should not be used it's more that data storage has a number of options that can be leveraged.  And some cases call for large databases to handle millions of transactions, but some don't. 

I don't have definite metrics on the systems, just that they are 24/7 systems with many real time feeds and expected response times.

Again, not belittling, just talking from development experience.
Bretttt Send private email
Tuesday, December 28, 2004
 
 
Crunch time comes when you have to handle concurrent access to data, and even within the RDBMS world every database is pretty much different (which is one of the problems with database agnostic developments). If you have to handle sessions modifying data while it is being read, or read while there is an uncommited change pending against it, then SQL Server and Oracle are going to behave [i]very[/i] differently.

But I wonder, even you don't have to worry about that then how much of the other code that is needed to replicate db functionality (indexing, integrity etc) has to be coded or bought in?
David Aldridge Send private email
Tuesday, December 28, 2004
 
 
I wouldn't say that it has to replicate DB functionality.  If you say that, then you're just writing another RDBMS.  You have to be able to handle the situations that you mentioned:  concurrent read/writes, speed, data/object identity. In any application you have to worry about these issues.  Many database to object mapping frameworks, though not perfect in the speed category, handle locking and integrity at the object level very well, and the object model is not tied down to a specific schema or db product for that matter.  And there are some good open source products too. 

Now I'm mostly looking at this from a developer's perspective.  Once, a large system hits prime time definitely need someone who can manage the different data stores effectively. But if you build the storage based on the object model then the support individual has to know how the objects interact instead of the other way around.  So it just depends on the paradigm you want to develop your system under.
Bretttt Send private email
Tuesday, December 28, 2004
 
 
>> Many database to object mapping frameworks, though not perfect in the speed category, handle locking and integrity at the object level very well, and the object model is not tied down to a specific schema or db product for that matter. <<

Ah, quite so, but aren't you then getting independence from one layer of your application by tying yourself in to dependence upon another layer? I think this is the issue at the heart of the myth of database independence - any kind of scalable data store has to have these functionalities, and if you make yourself independent of the RDBMS then you are just making yourself dependent upon a different (and less mature) component.

There is also the performance issue of course ... "database independence" for any kind of complex data storage application generally means poor performance, and poor performance means high support and hardware costs. It's a vicious circle -- in order to be able to change the RDBMS "backend", techniques are used that lead to poor performance, and that leads to questions over the value-for-money of the RDBMS, and that leads to RDBMS-hopping in search of a cheaper/more vanilla solution?
David Aldridge Send private email
Tuesday, December 28, 2004
 
 
"then getting independence from one layer of your application by tying yourself in to dependence upon another layer?"

I think it's better stated that you'd be consolidating business functionality and data structure. A problem I have with our Sybase setup is that from a DB perspective I don't have a way of accessing remote servers through SQL.  Sure separate DBs but not other servers.  Well, this is definitely not a problem when the different server's databases are brought under one abstract framework and under one language and into one virtual machine.  And then you have all of the functionality with the data in one very scalable/malleable environment.
Bretttt Send private email
Tuesday, December 28, 2004
 
 
You can take the piss out of 'tight-arsed' database people all you like. And make fun of their little tantrums about wanting to keep tables normalised. It's easy, they aren't hip to the latest hot programming languages. After all, they only look after boring data. Some of them wear suits even. And I've never met a DBA with a single body piercing! They're just like so, square Daddy-O.

Bah!. Stuff and nonsense! Baloney! Balderdash! Tommy rot! If the application you as a programmer are working on is a database then the database rules. FULL BLOODY STOP.

Loops And Conditions Monthly (incorporating Goatees for Devs) may suggest we all change to C#/XML/eXtreme Programming or whatever. I don't give a toss.

As far as I can see the relational database is the best we've got. It is a method based on sound mathematical and logical principles to ensure that data are accurate and yes, USEFUL, to the interested parties. Scalar values, one datum stored once, each entity instance identifiable etc. are just so obvious as basic principles. It makes sense.

You either accept that or else write your own data storage mechanism. Yeah, flat files. Go on, do it.

The RDBMS is the centre. Programmers work round it. If it's too hard, tough.

Of course, that's just my opinion. Wouldn't like to appear intolerant or anything
Database pedant
Tuesday, December 28, 2004
 
 
I have spent my entire programming carrer working around data specific aplication developement and enhancement.  Ill admit it has only been 7 years but its what I have done for the entire seven years.  As for size it has been around 200 or so tables with 10 or so topping 25 millon rows each.  For volume roughly 30 thousand online transactions a day from upwards of 1000 users with batch volume closer to the 200 thousand mark.  That is for updates/adds/deletes. I really never look at a metric for the reads for the dozens of reports produced at user request and on a regular cycle.

  A good DBA can be a great asset or the worst hinderance and even kill projects.  You would hope that a good DBA knows the tricks for performance but will have little idea of how and what the data will be used for.

(I removed alot here as to answere the question and not rant you only need whats left)

In short the DBA is only part of the puzzle that knows the database not the actuall data or its use.  You must be willing to work with others and know your limitations.  If you cannot build the stored procedures timely then train and mentor some of the better developers so you can trust them to lighten that load.  Work with those people that need to consume the data and its ultimate purpose to be able to accuratly help in the process and not burdon it with what may be an elegant database design that is completly useless for needs of the data to be stored.
Douglas Send private email
Tuesday, December 28, 2004
 
 
What does a 'DBA' do?
Database pedant
Tuesday, December 28, 2004
 
 
I only ask because you seem to think the 'DBA' just exists to tinker about and write the 'stored procedures' that presumably you want writing to satisfy your users information needs. Write 'em yourself. It can't be hard. Oh and if it's a bit on the slow side could the DBA just sort of fiddle with it a bit to make it faster please.

These pearls I love:

'You would hope that a good DBA knows the tricks for performance but will have little idea of how and what the data will be used for.'

'In short the DBA is only part of the puzzle that knows the database not the actuall data or its use.'

That's it. If you're representative of the mindset of the programmers working on database applications then we're stuffed. Those 2 statements are about the dimmest things I've ever read from someone who works with databases. To say you have missed the point is an understatement on acid.

The business of database design is centred on the fact that we DO know what the data are (notice ARE, it's a bloody plural, plural of datum, 'known fact') for. The essence of database design is semantics. We should know the MEANING of the data. And only then is it possible to design a database. It is not and cannot be a mechanical process. We don't just come up with a bunch of tables so the Java whizzes (or whoever) can work their magic.

Do yourself a favour. Get a copy of CJ Date's An Introduction To Database System's, read it and understand it. Of course it'll be hard to find. But it'll be there somewhere, behind 'XML is the answer' and it's like.

God help us
Database pedant
Tuesday, December 28, 2004
 
 
DBA: "your application provides editing services for my database"

Developer: "Your database provides persistence services for my application"

Both are right. But a a developer, I don't want to have to know anything about the database. I just want the simplest way of having my data avialble at any time. Ideally with no API. That's why there is so much parallel development of O/R mapping tools/APIs/Frameworks. I don't wwant to have to know about them either. This is most appealing: http://www.prevayler.org/wiki.jsp

A python version is on the way ;-)))

Persistent re-useable data is critical. The new generation of DBs are providing Relational (SQL), Object and XML interfaces; and even native XML and Object storage. Cool. This is better for me and makes DBAs even more valuable.

So many Enterprise apps are just table editors and report writers and they look like table editors and report writers. This is what I think of when I hear "Data-driven" application; which really mean "Database driven application"; which really means "Relational database driven application"; which really measn "SQL database driven application".

Static web sites are data driven as are all applications that persist data.

I want to delete SQL from my memory. It is simultaneously the best and worst thing IBM ever invented.
fool for python
Tuesday, December 28, 2004
 
 
I think that Douglas has touched again on one of the major issues -- staffing. But it's not only hiring the right people, it's putting them to work in the best way.

Now, as I said before I've met DBA's who are absolutely useless at actual database programming (or even SQL), and that's because they are really two different skill sets (except the SQL -- that's essential of course). I wouldn't hire a DBA to write an application, because 90% of them just do not have the skills for it -- it's like expecting a unix sysadmin to be a competant C programmer. Sure it would help, but it's not part of their core skill set.

I think that if I was staffing the development of an information system then I'd want dedicated & skilled database programmers, who know what subquery factoring and analytic functions are, and who know the pros and cons of implicit and explicit cursors. Unfortunately they are like gold-dust to find and hold on to, and I very much doubt that a modestly sized software development company could find and keep them. I'd be willing to tie myself into a database vendor also -- 99% of the rest of the application is tied in to some technology or vendor anyway.
David Aldridge Send private email
Tuesday, December 28, 2004
 
 
'the primary job of DBA's is to say "no".'

That's my experience of the people who are hired as a "DBA" rather than being the developer who specialises in learning how to tune the DB.

Typical conversations would run "Hey, we need a new table creating. We've got this project due in a month."

"Yeah, you see, we can't just run around creating tables. There'd be chaos. We'll have to evaluate the impact. It'll take five weeks to schedule an evaluation conference.."

The end result is that it's impossible to get tables added. Ever.

"The Business" is never interested in applying any pressure to make the DBAs do any work ever, so developers end up trapped between a bunch of DBAs who just say no, and a business that won't take no for an answer. And lo and behold, sanity gets squeezed out in the middle.

The result of that is that there are all manner of bizarre work-rounds, from using other tables which are "hanging around spare" to mission critical data being shoved in regular files on a server somewhere (which of course aren't backed up).

And I'm not talking small businesses here. I'm talking banks, logistics companies, global shipping companies, financial exchanges....

Dedicated DBAs basically just get in the way and anyone who hires one is 99% promised a death warrant for their project having a sane design. Asking them to help with a project is like asking HR to help. The verbosity of the reply will change from time to time, but not the usefulness of it..

The few DBAs who take an interest and turn up to the development meetings are amazingly useful. They'll look at designs and go "WHAT ARE YOU DOING???? Look, I'll write you a join that'll solve that problem without all that code and overhead..." and they contribute. They're stunningly rare and undervalued and you can't hire them for love nor money.

Wednesday, December 29, 2004
 
 
This discussion, and one's like it, just appalls me. Well, saddens really. Here we are in a forum where people will natter on for ages about whether this programming tool or that is better for task A or B. Who prefers this code editor because of intellisense/indenting style or whatever. If only the same depth of discussion happened about the base of pretty much all we do, the database.

Let's face it, most 4GL languages are pretty similar. One Do While loop is much the same as another, all you have to do is learn the new syntax. The extent of object orientation may well be an issue, the type safeness another, platform specificness another, I agree. There are choices to be made.

But the incredible lack of understanding of database theory, of how fundamental it is AMONGST IT PRACTITIONERS is appalling. Hands up those of you who work on non-database applications. Hello? Right, thought so. The vast majority of applications are databases, aren't they? So why are people so casual about it.

It seems that the attitude is 'chuck us a bunch of tables and we'll do the real work'. 'It's so hard to get the DBAs to create a new table'. WWWHHHAAATTT?. You, the programmer have decided you need a new table so the DBA better get on and make one? Who exactly is the code monkey here?

I used to think that Fabian Pascal was overly aggressive in his columns. Having seen so much sloppy talk FROM DEVELOPERS about databases I can see why he's like he is.
The database guerilla
Wednesday, December 29, 2004
 
 
Prevayler: What's the story with those benchmarks? Are they saying that Oracle will only manage between 9 and 15 simple lookups per second?

I notice this: "The tests are limited to very simple operations so that the relational databases can cope". Ha ha ha. My database can join 20 tables together very nicely, thanks a lot.

Anyway, more constructively I can guess what they're doing -- they're not using bind variables, and I must say that's a very typical problem when non-db programmers use Oracle.

As an example I created a 160,000 row two column table in Oracle 10g, and queried it to return about 20 rows through a non-unique index scan. The first time it ran, the parse (including optimization) took about 2/100ths of a second -- every subsequent time it effectively took nothing.

So, here was my own benchmark -- perform 10,000 single row lookups from an index in Oracle 10g. Total elapsed time: 0.3 seconds. The Prevayler benchmarks look like a real pile of crap to me.
David Aldridge Send private email
Wednesday, December 29, 2004
 
 
Heh heh heh -- I don't think Fabian Pascal's problem is with developers per se, it's with the entire database-related industry (and academia). I like his website. It's like watching someone make a career from passionately advocating the global replacement of English with Esperanto -- admirable aims, but wholly impractical.
David Aldridge Send private email
Wednesday, December 29, 2004
 
 
>> But the incredible lack of understanding of database theory, of how fundamental it is AMONGST IT PRACTITIONERS is appalling. <<

Yup, and not just the theory, the practicalties too -- Oracle developers (and DBA's) who have actually read the "Oracle Concepts Guide" and understand the architecture are one-in-a-thousand.
David Aldridge Send private email
Wednesday, December 29, 2004
 
 
Perhaps. If FB (er, that's Fabian Pascal) serves no other purpose than to point out the paucity of knowledge amongst those who should know better, then he's doing a good job.

And to keep pushing at the boundaries of relational database technology, like the 3rd Manifesto stuff, is good.

The casualness with which people discuss these matters is what's scary. After all:

Select Case int
 Case 1
  Crash the computer
 Case 2
  Declare 3rd World War
End Case

would be considered extremely sub standard ('nope, that integer will never ever ever be anything but 1 or 2'). Hint: it needs a case else.

Yet the same rigour in database design is viewed as some sort of 'up-tightness'.
The database guerilla
Wednesday, December 29, 2004
 
 
Maybe the main problem is that crap database design doesn't generate compile errors.
The database guerilla
Wednesday, December 29, 2004
 
 
I agree with the sentiment -- there are database projects out there that really do a proper job of their developments, and there are many more with fundamentally flawed practices. However I don't see FP as an advocate of good development practices -- he's an advocate of "drop SQL and all current RDBMS implementations, and invent and adopt a strictly relational implementation". there's nothing there that you or I can use to make our own developments better in any way at all.
David Aldridge Send private email
Wednesday, December 29, 2004
 
 
Yes, I know. He seems to want us to sit around waiting for a TRDBMS to arrive.

I would say that despite all his curmugeonliness (?) - a style I try to emulate myself - he's one of the few people keeping standards up. I waddled through umpteen Access/SQL Server books, dutifully reading the half dozen pages about 'database design' as though that was enough. It was only when I actually stopped thinking about implementation and looked at the basic Requirements to CDM to CORRECT logical model process that I became anything like a database developer.

So although there is a great deal of ivory towerness about FP, most of what he says can be applied to our (in his opinion) current less than perfect tools.

Problem is I get the distinct impression that many 'programmers' didn't even read those half dozen pages.
The database guerilla
Wednesday, December 29, 2004
 
 
>> Problem is I get the distinct impression that many 'programmers' didn't even read those half dozen pages. <<

Amen. And if the database practitioners are generally not competant enough with their own technology, then non-db application programmers can hardly be blamed for making mistakes. Oh, well actually they can be blamed, and so can the poor quality db professionals. ?I don't think that the answer is to avoid RDBMS's though, it's just to hire the right people.
David Aldridge Send private email
Wednesday, December 29, 2004
 
 
As to the question of what a DBA is.

I have personally never been one but it has always appeared that the DBA was responsible for the database itself.  Insuring patch updates are installed that is available on the servers needed.  To monitor and tweak size allocation as well as disk distribution of its tables.  The security accesses to the databases under their care.  I am sure I am missing several things here like backups and restore.

The comment that the database layer of the application is often trivialized seems on the mark from what I have experienced.  In the corporations that I have worked in management in an attempt to fix this assigns responsibility to those who I assume they feel it makes sense to the DBA's.  From there either threw database security and/or policy the application developers are excluded from creating or modifying stored procedures or tables.  This of course is without increasing the staffing of them.

Personally it needs to be a partnership between the application developer and the Database Team.  I change to the term Database team simply because somewhere you need someone with expertise with the database engine from its language perspective and I don’t think that is truly the realm of the administrator.  The developers as well need to get their act together and get some database fundamentals at least under their belt when they work with it.  The only reason I hesitate to put the responsibility of the design and usage fully onto the developer is that at least in my experience most projects seem to span multiple data storage options.  If the corporation is only going to be using one database engine then I can see the developer getting off their collective duffs and learn what they should to efficiently and correctly deal with the database.

Sadly it has been my experience that most of the developers take the attitude that they should not and will not care about the database handling.  Shoot most I have worked with tools that allow them to tune their SQL statements.  The last time I was on an SQL database was Sybase and we were given tools that allowed us to see how the statements where being implemented by the engine.  I liked it as it gave back performance metrics as well as when it was using a key and when it was not.  This to me allowed you to tweak the statements around for the best performance you could get.  On the DBA side I have seen them forced willingly or otherwise into becoming partial programmers and not just administrators by management forcing them to create all tables and all stored procedures threw a request process that isolates them from the actual need and purpose of the data to be stored or the stored procedure for it.

Someone mentioned that if I wanted a stored procedure as a developer I should create it myself.  My response to that is I WOULD F'ING LOVE TO, but alas I do not have the security access to the database to do so and am bared from even attempting to by corporate policy.  The best I can do is fill out a form and try to fit in the actual need into a form that was designed to specifically exclude such needless information and wait and hope its creation is approved and then finally created in time for the implementation.

Ideally to me in a perfect environment let the DBA administrate the database let me take the responsibility to maintain my table structure and stored procedures.  At least on my personal projects I wind up with reference manuals for general database design as well as the particular database engine I am using right along with references for the programming language or languages I am using.

But with the seeming movement to isolate the application from the data layer there is a gap that needs to be filled somewhere.  Is it the DBA that needs to step into this role or some sort of Data Architect or some such is a question I fell unqualified to answer at this stage without a ton more research and contemplation than I have currently given it.
Douglas Send private email
Wednesday, December 29, 2004
 
 
Yes, I think you're right with regard to the DBA role -- most often I've herad it said that the top three priorities for a DBA should be ...

* Backup & Recovery
* Backup & Recovery
* Backup & Recovery

Now in recent years there has been a move in all RDBMSs towards simpler management, and in Oracle's case this makes it much easier for (relatively) less sophisticated persons to create database objects robustly -- for example by not needing storage parameters to be specified for tables and indexes, just put all tables in one, or maybe one of three, tablespaces. On the other hand, I still couldn't bring myself to just let folks create and modify tables on "my" database without some kind of oversight -- there are issues of naming conventions, correctness of data types, correctness of table structure (heap, IOT, clustering etc) and index types (b-tree, bitmap, reverse-ordered etc) to be considered. Doesn't have to be a big song-and-dance over it, or a lengthy review process. Just a quick check for the application of best practices. How many of us takes care to use a reverse-byte index on a synthetic numeric primary key column, to avoid having to trace locking problems on concurrent insert of new rows when the application goes into production? Very few, that's how many.

There's also an interesting issue raised by two points here ... "The DBA won't let me just create a new table" and "The database guys don't use proper source control". I'll let them speak for themselves.

And you also point out the familiar organizational and staffing issues. IIWG (If I Was God) there would be no such thing as a database team, just an affiliation -- it's just a technical skill no different to being a Python/Java/unix person. When a team is formed to develop a system, then those with database skills go in with the rest. I hate that Chinese Wall crap that tends to develop between db and "application" people.

Oh well, back to work.
David Aldridge Send private email
Wednesday, December 29, 2004
 
 
"You, the programmer have decided you need a new table so the DBA better get on and make one?"

No, you've convinced me. I shouldn't go around asking for extra stuff to be put into the databases, being a mere applications developer.

All you have to do now is get the business people to agree to stop asking ME for new features.

Then we can BOTH sit on our backsides all day. We don't ask for stuff to be done for FUN. We ask for it to be done because we need it done to meet someone else's needs. And those needs would come from people in the business. The ones who pay for all this.

This is what I mean about unco-operative DBAs. You're like HR. HR also thinks the company exists to serve them, instead of the other way round.

Wednesday, December 29, 2004
 
 
The lack of understanding about the "DBA" is amazing here. There are all sorts of jobs with the title of DBA. Some managers say DBA when they mean database developer (writing stored procedures, functions, etc.). Some mean Server Administrators of Database Servers (Backup/Restore, hardware tuning), some mean Data Architect (Data modeling, schema change, database design), and some mean all of the above.

This is like us arguing what is actually a developer. Code monkey, Architect, Application administrator (with little or no actual code writing), etc. Some or all of the above, depending on where you work and what the boss tells you to do.

I am sure that there are "developers" that are just as harmfull to a project as some of the overzealous DBAs categorized here. The big thing you have to remember, a true DBAs first allegiance is to dat integrity and recoverability. Just asking for a table beacuse you got a request does not contribute to these integrity or recoverability. Propper database design, change manegement, version control, etc. do.
anomalous
Wednesday, December 29, 2004
 
 
"But the incredible lack of understanding of database theory, of how fundamental it is AMONGST IT PRACTITIONERS is appalling."

Yes, exactly the problem. Application programmers should not have to be database practitioners. They should not even have to know much database theory. They should just use simple persistence APIs and let the DB theorists and practiotioners  worry about what happens behind that. One of the largest databases out there is the web. HTTP provides a few simple verbs and with http://someurl I can get at it. Is there SQL at the other end? Probably, but that's a job for the web application framework folks and database designers; NOT application developers.
fool for python
Wednesday, December 29, 2004
 
 
"a true DBAs first allegiance is to data integrity and recoverability"

Nooooo. A true DBA's first allegiance is to do whatever he's told to do by his bosses. You can try and convince them of the wrongness of their ways, but using belligerance and intransigence to "get your own way" isn't honest. Your own oracle server at home -- you admin that the way you want.

The one at work belongs to your bosses.


{If they ask you to do something illegal, that's different.}
Katie Lucas
Wednesday, December 29, 2004
 
 
Adam Bosworth has some thoughts on current DB offerings:

http://www.adambosworth.net/archives/000038.html
fool for python
Wednesday, December 29, 2004
 
 
'"a true DBAs first allegiance is to data integrity and recoverability"

Nooooo. A true DBA's first allegiance is to do whatever he's told to do by his bosses. You can try and convince them of the wrongness of their ways, but using belligerance and intransigence to "get your own way" isn't honest. Your own oracle server at home -- you admin that the way you want.

The one at work belongs to your bosses.'

So you do it the bosses way, the server crashes and you lose data. That's OK is it? When the 'bosses' come screaming at you and you say, well I did what you asked and look what happenned. You know as well as I do that it'd be 'you're the professional you should be able to sort this out'.

A database person isn't employed to blindly follow the orders of their bosses. If it was that simple then the bosses could do it all themselves. They are employed to use their skill knowledge and training in the service of their employer. And if that involves saying 'no' sometimes then they're doing their job.
Unemployed DBA
Thursday, December 30, 2004
 
 
'Adam Bosworth has some thoughts on current DB offerings:

http://www.adambosworth.net/archives/000038.html '

Including this gem:

'It is this. Users of databases tend to ask for three very simple things:

1) Dynamic schema so that [snip] In short, the database should handle unlimited change.'

The snip doesn't alter the meaning here. Of course we try to design databases so they can accomodate change. THAT'S THE POINT OF RELATIONAL DATABASES. Well, part of the point. But the idea that they should be able to handle 'unlimited' change is just ridiculous. 'We don't know what our information needs will be in the future, but the database should be able to cope with it'. Oh, so database designers have to be clairvoyant now?

Or this:

'In short, the database should handle unlimited scale with very low latency.' Look Adam Bosworth, lose the unlimited word will you. There are limits to everything.

He complains that database vendors aren't doing what he wants, oh yes, Oracle is having a good go but they charge too much and the free OSS people aren't doing it either.

Do it yourself Adam. Write your own cheap, unlimited scalability, unlimited changeability, 24x7x365 DBMS. You'll clean up. You can give it to programmers for nothing. Even better if it can work without any awkward database person to get in the way.
Unemployed DBA
Thursday, December 30, 2004
 
 
"Oh, so database designers have to be clairvoyant now?"

No, databases have to be more flexible. That's all. And more flexible, more scalabel DBs should increase employment opportunities for DBAs.
fool for python
Thursday, December 30, 2004
 
 
Well do it. Stop complaining that they aren't flexible enough and do it yourself.

The vast majority of my work at the moment is complete life cycle development for SMEs. So I do the analysis, design, data schema and interface. So I know how much longer it takes to change something if you don't get the data structures right in the first place.

Even better, I can't complain about instransigent DBAs. Cause it's me. And I'm the programmer too.
Complete life cycle guy
Thursday, December 30, 2004
 
 
I'll never forget the sigh of relief i heard from one programmer when they used JDO instead of hibernate. From an application writers viewpoint having to deal with all the crap of hibernate and other OR mapping layers is horrible.

If you can get away with working at just the object layer then life isn't so bad. But as long as you have to deal with a multi-application corporate repository all these issues won't ever go away.
son of parnas
Thursday, December 30, 2004
 
 
>>Bah!. Stuff and nonsense! Baloney! Balderdash! Tommy rot! If the application you as a programmer are working on is a database then the database rules. FULL BLOODY STOP.<<

FULL BLOODY STOP is right. The database does not rule...or, I should say, it should not rule. The ONLY thing that matters or should matter in the process of designing any system or database is the end user's end experience. End of story.

Perhaps I am hopelessly out of touch and pitifully under-educated. Perhaps I am out of touch with current technology (this is completely possible, even probable) and behind the times but I simply cannot understand any useful much less outstanding application coming out of a separation of the design/development/future modification of that application and the design/development/future modification of the database feeding into it.

The process of normalization is something I do with all databases I design. Then I denormalize as needed to provide increased performance/functionality to the application I am designing. It's a give and take process resulting, hopefully, in the satisfaction and fulfillment of the needs of the end user's.

The definition of DBA seems to vary greatly depending on the company they work for, etc.. Apparently, in some instances, the title DBA is responsible for deciding what tables and relations are necessary in any given database they administer. I am completely lost. How in the world could a DBA in this context (acting as a separate entity, being only a DBA and not an analyst/programmer) know what tables/relationships are necessary to provide speed and functionality to a user without knowing all the results of a needs analysis. I can't imagine designing a system and not also designing the database and changing it when necessary based on new user requirements gleaned through system analysis functions and the resultant necessary code changes. I can't imagine designing and developing an end user system without fully understanding and knowing how to implement a database back end.

How does this work out there? A user has a new requirement for information that they want and need out of the system. You modify the design of the system to provide that new information to the user. You realize this will require a change in the database structure in order to give the user the best possible solution to their needs. But wait...Ooops...can't do that cause a separate person who is only in charge of making sure the database remains normalized and intact says you can't, because the "DATABASE RULES".

On an enterprise level, say you want a data warehouse concept so that many different departments can access a central database of information based on each of the department’s unique needs. Fine. Same question holds. A collaboration of developers needs to occur and a central project leadership role needs to be filled but I still don't get how someone who does not do the needs assessment (a system analyst) and development (analyst/programmers) can possibly design and control creation of and changes to a database when the uses of the data by the end user are unknown.

Or maybe DBA's in most organizations are also Analyst/Programmers. Or maybe all System Analysts and programmers have been reduced to code monkeys in today's work world and consultants are hired to do needs assessments.

I currently(hopefully very temporarily) work for state government where all things are surreal and am developing niche market total business management software at night. I haven't worked for private industry IT departments for about 10 years or so. However, when I did work for private industry this is not the way it worked.

I'm just looking for an explanation of how development and database design can be separated? Anyone can normalize a database. How can you do this without knowing how it will be used? In reverse, how can you design a system without controlling the underlying database?
Deborah Miller Send private email
Sunday, January 02, 2005
 
 
>> I'm just looking for an explanation of how development and database design can be separated? Anyone can normalize a database. How can you do this without knowing how it will be used? In reverse, how can you design a system without controlling the underlying database? <<

If we accept that the user interface, data storage, and application logic are all part of a single system design (which I think we all do) then that still doesn't tell us anything about where the application logic should reside, no? Is it not the "natural" arrangement that the data and the data manipulation logic should reside in the same layer? (aside from cases here multiple incompatible data stores are an unavoidable evil)

For example Oracle's PL/SQL was designed for efficient data manipulation, and is continually updated with this in mind. PL/SQL developers are generally going to be "closer" to the dreaded DBA, and better equipped to argue the case for more/less normalization etc..

Remember that we're not talking about trivial single-user systems, where you don't have to think about concurrency, data integrity, locking etc..
David Aldridge Send private email
Monday, January 03, 2005
 
 
'The process of normalization is something I do with all databases I design. Then I denormalize as needed to provide increased performance/functionality to the application I am designing.'

I've been following this. Perhaps you could give me an example of something you did recently where you had to denormalize to achieve performance or functionality?
Mike MacSween
Monday, January 03, 2005
 
 
>> Perhaps you could give me an example of something you did recently where you had to denormalize to achieve performance or functionality?

... and then, get the asbestos underpants on!
David Aldridge Send private email
Tuesday, January 04, 2005
 
 
First of all, data is singular, not plural. The people who wrote the dictionary were wrong. We speak English, not French or Latin; English has a collective noun construct. Would we ask girls what color their hair are today? What are the news today? En francais, c'est parfait, mais on ecrit ici en anglais...

Anyway, I never understood why people wrote for either the front end or the back end, but not both. I've always wrote for both sides of the code, which leads to much better code on both sides.

A nice bene is that I spent no time mapping front end structures to relational structures. When you create them both at the same time, that silly task goes away.
Stephen Hirsch Send private email
Tuesday, January 04, 2005
 
 
Slightly off-topic, but brought up from before:  What IS the difference between "SELECT COUNT(*)" and "SELECT COUNT (my_column)"?

I've looked for this before, but Google is singularly un-helpful.  I always assumed that COUNT(*) would pull all records from the table, which could be disastrous if you had large text blobs, so I use COUNT(my_column) on the off chance I'm right.

But is there a functional difference as well?  As in, will they return different values if say, my_column has nulls in it or somesuch?
Dan Hulton Send private email
Thursday, January 06, 2005
 
 
Dan,

Count(*) is "count all rows in the table".

Count(my_column) is "count all rows in the table for which my_column is not null".
David Aldridge Send private email
Thursday, January 06, 2005
 
 
I should add that any RDBMS worth it's salt would not necessarily read the entire table in order to calulate Count(*) -- the count can be based on an index guaranteed to have no null entries in it, and 99% of database tables (particularly large ones) will have such an index available.
David Aldridge Send private email
Thursday, January 06, 2005
 
 
Yeah, I always use COUNT(key_field), which is guaranteed to not have any null entries in it.  So I suppose the difference to me between the two is null.  But it is good to finally know.
Dan Hulton Send private email
Friday, January 07, 2005
 
 
What RDBMS is that on, Dan? Just as an aid to maintenance I'd stick with Count(*) myself, and let the optimizer choose which index to scan.
David Aldridge Send private email
Friday, January 07, 2005
 
 

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics
 
Powered by FogBugz