Topic: Canadian Government Call for OSS

[Bonk]

Federal government puts out a call for information on free software

A little late - only two days left - but there it is. Thank god.

Problem is, I can't find where to make said submissions on the Merx site. If anyone has any luck finding it, let me know as I want to be sure that MySQL and PostgreSQL get recommended.

Edit: ah, here it is:
       RFI - NO CHARGE LICENSED SOFTWARE

It would be nice if there were some way to see what has already been submitted.

Though one of the commentors on the cbc story points out these links:
http://software.pwgsc.gc.ca/catalogue/index-e.cfm?attributes.fuseaction=prod-category&CategoryID=1345
http://software.pwgsc.gc.ca/catalogue/index-e.cfm?attributes.fuseaction=index
and questions why that existing process is not being used. (I think I know why, I expect someone has been asked to write a report, this is a good way to get material)

[Nemesis]

I think that governments and companies would do better concentrating on open data formats so they don't get locked into one supplier. 

With open formats those dealing with the company/business don't get locked in to having to buy and use the same software, they can use their choice so long as it is compatible with the open format.

[Bonk]

Agreed, and I think that is recognised but is not as much a concern as licensing costs. It is also not as much of a concern these days as many OSS applications do a reasonably good job of importing proprietary formats. Additionally, most new proprietary software is pretty good about providing export capability in standard formats (though it might not export all features of the native file format).

I'm thinking in particular about GC/LC-MS data, the scientific community generally requires interoperability of data files, at a minimum for communication and publication purposes. (e.g. NetCDF) However almost all data acquisition systems have their own native file format that performs best, naturally.

But why I am really a proponent of open source software in government and science is essentially accountability. The validity and logic of open source software is open to scrutiny, which more and more I beleive is a must for good science. Part of my reasons for advocating OSS in government and science is adherence to standards as you suggest. The critical concept for me though is validation. How can I validate software when provided only with a binary? Sure, it works this time I test it, but how do I know it will always behave that way? (think Y2K, but there are so many other variables than just time).

[toasty0]

I think that governments and companies would do better concentrating on open data formats so they don't get locked into one supplier. 

With open formats those dealing with the company/business don't get locked in to having to buy and use the same software, they can use their choice so long as it is compatible with the open format.

XML is used for open or non-vendor specific data transfer. I can move data from an Oracle db to SQL to Sql to MSAccess using XML with just about schema I want.

I'm pretty sure that most--if not all the players--have agreed on a virtualization spec. Now, something virtualized on VMware will work on Hyper-V and reverse. I'm sure there are still kinks, but things have come a long way from 5 years ago.

[Nemesis]

I think that governments and companies would do better concentrating on open data formats so they don't get locked into one supplier. 

With open formats those dealing with the company/business don't get locked in to having to buy and use the same software, they can use their choice so long as it is compatible with the open format.

XML is used for open or non-vendor specific data transfer. I can move data from an Oracle db to SQL to Sql to MSAccess using XML with just about schema I want.

I'm pretty sure that most--if not all the players--have agreed on a virtualization spec. Now, something virtualized on VMware will work on Hyper-V and reverse. I'm sure there are still kinks, but things have come a long way from 5 years ago.

I have to disagree.  With a standardized format it shouldn't matter what program you use to read/write it.  That allows you to change programs without having to "translate" all your files. 

For companies and government agencies that can easily run into the millions of files.  Translating and checking all those files is a substantial cost.  A cost that can be avoided by using a standardized format.

[toasty0]

I think that governments and companies would do better concentrating on open data formats so they don't get locked into one supplier. 

With open formats those dealing with the company/business don't get locked in to having to buy and use the same software, they can use their choice so long as it is compatible with the open format.

XML is used for open or non-vendor specific data transfer. I can move data from an Oracle db to SQL to Sql to MSAccess using XML with just about schema I want.

I'm pretty sure that most--if not all the players--have agreed on a virtualization spec. Now, something virtualized on VMware will work on Hyper-V and reverse. I'm sure there are still kinks, but things have come a long way from 5 years ago.

I have to disagree.  With a standardized format it shouldn't matter what program you use to read/write it.  That allows you to change programs without having to "translate" all your files. 

I think I may have missed your point. XML is an open, non-vender specification. http://www.w3.org/XML/Core/#Publications What is about XML that is not meeting your needs?

For companies and government agencies that can easily run into the millions of files.  Translating and checking all those files is a substantial cost.  A cost that can be avoided by using a standardized format.

For data XML fits the bill. Can you tell me what I'm missing?

With XML, now anyone can use any software they want and the tansfer of data and data schemas between users should be tranparent and seemless.

[Nemesis]

XML is used for open or non-vendor specific data transfer. I can move data from an Oracle db to SQL to Sql to MSAccess using XML with just about schema I want.

I'm pretty sure that most--if not all the players--have agreed on a virtualization spec. Now, something virtualized on VMware will work on Hyper-V and reverse. I'm sure there are still kinks, but things have come a long way from 5 years ago.

I have to disagree.  With a standardized format it shouldn't matter what program you use to read/write it.  That allows you to change programs without having to "translate" all your files. 

I think I may have missed your point. XML is an open, non-vender specification. http://www.w3.org/XML/Core/#Publications What is about XML that is not meeting your needs?

For companies and government agencies that can easily run into the millions of files.  Translating and checking all those files is a substantial cost.  A cost that can be avoided by using a standardized format.

For data XML fits the bill. Can you tell me what I'm missing?

With XML, now anyone can use any software they want and the tansfer of data and data schemas between users should be tranparent and seemless.

I don't claim any real knowledge of XML but from what little I do know it doesn't count as a format. 

With a standard format the new program would just be able to open and edit the file without any processing.  You don't risk conversion errors and you don't risk the problem of compounding those errors by translating back and forth over time. 

As I understand what you wrote about moving a database  you are using XML to translate between the native formats of various databases you mentioned.  They don't just open and use each others files.  There is a translation process and there should be a verification process as well to ensure the data migrates seamlessly. 

So long as the formats in use are different there will be a translation effort and that will impose costs on moving the data and possible errors introduced in the translation between formats. 

With a standardized format I could E-Mail you a file from my word processor using the standard and you would just be able to open it without any knowledge of what program it was created in and edit it without issue.  Then you could mail it on to someone else who could open it in a 3rd program without issue and so forth.  Can you do that with XML?

A question.  If XML makes things so easy why is it that Microsoft does not plan to support OOXML (which is a derivative of DOCX) until the next full version of MS-Office is released?  Why is it that no one yet supports OOXML?

[toasty0]

I think that governments and companies would do better concentrating on open data formats so they don't get locked into one supplier. 

With open formats those dealing with the company/business don't get locked in to having to buy and use the same software, they can use their choice so long as it is compatible with the open format.

XML is used for open or non-vendor specific data transfer. I can move data from an Oracle db to SQL to Sql to MSAccess using XML with just about schema I want.

I'm pretty sure that most--if not all the players--have agreed on a virtualization spec. Now, something virtualized on VMware will work on Hyper-V and reverse. I'm sure there are still kinks, but things have come a long way from 5 years ago.

I have to disagree.  With a standardized format it shouldn't matter what program you use to read/write it.  That allows you to change programs without having to "translate" all your files. 

For companies and government agencies that can easily run into the millions of files.  Translating and checking all those files is a substantial cost.  A cost that can be avoided by using a standardized format.

So--and this might H&S material--what you're proposing is that government decide what the format should be?

[Nemesis]

So--and this might H&S material--what you're proposing is that government decide what the format should be?

It isn't political in any way so I don't think it is really H&S material.

I want the government to use open standards where ever possible so that there can be competition both in software sales and service to them and to encourage competition in the market as a whole.  I'm not saying that they should establish a standard but to encourage standards and choose those standards that they will use rather than getting locked into a proprietary format and one software supplier.

When dealing with proprietary formats once the government chooses to use one that software company has a lock on the contract as effectively no one else can bid on it.  All your government data in that format is only accessible by software from one supplier.  If the government chooses to change software suppliers they have to factor in the enormous cost of converting all their existing data.  This works to keep the incumbent software provider in control as the competition has to under price them by enough to make up for the data conversion and verification costs.

From the government you get ripple effects as to exchange data with them you have to use the same format and if it is proprietary that means using the same software.  So companies bidding on other government contracts have to use software from the same supplier as the government just to be able to submit bids.  Consider the complaints post Katrina when people could only access government aid sites using IE, they locked out anyone using non Microsoft operating systems and those using old versions of Windows that couldn't run a new enough version of IE.  Using standards would avoid this. 

This data compatibility/lockin extends into business as well.  My former employer after a corporate takeover spent a fortune changing all of our data from Word Perfect to MS-Office as that is what the new owners used.  Most of that money would not have needed to be spent if they were using a standardized format as they would have just had the cost of buying the new software not the data migration costs.

When it comes to exchanging data between companies you have the same issue as with the government.  An acquaintance of mine is part owner of a small business with one secretary.  They had an opportunity to get a large (from their viewpoint) contract with GM but they had to use MS-Office file formats for the data exchange.  To get the contract they had to buy MS-Office and get their secretary retrained.  Not a big thing for a large company but for a small company like theirs it was a major issue.  Especially since they had to either maintain two separate systems for data stored in the different formats or convert everything over to MS-Office. 

Consider other standard formats they you most likely use and how it would affect the market place if each company had their own.  CDs, and DVDs, are two such.

[toasty0]

I'm still confused why you think the XML standard is contrary to that doctrine. All of the software I mentioned (SQL, Oracle, Sql) all seemlessly export and import data from and to each other using the XML specification.

[Nemesis]

I'm still confused why you think the XML standard is contrary to that doctrine. All of the software I mentioned (SQL, Oracle, Sql) all seemlessly export and import data from and to each other using the XML specification.

You just said the answer IMPORT and EXPORT.  They are using different formats and translating back and forth adds a great deal of chance at creating errors.   You don't just load your Oracle database into MS-SQL do you ?  You have to convert the format to the MS version from the Oracle version.  You are adding two steps, convert and verify (or convert and hope it works if you don't verify).

Ever seen something translated out of English and then back again?  How many changes were there from the original?  I read just enough French to see that quite often the French and English versions of things are often subtly different.

[toasty0]

I'm still confused why you think the XML standard is contrary to that doctrine. All of the software I mentioned (SQL, Oracle, Sql) all seemlessly export and import data from and to each other using the XML specification.

You just said the answer IMPORT and EXPORT.  They are using different formats and translating back and forth adds a great deal of chance at creating errors.   You don't just load your Oracle database into MS-SQL do you ?  You have to convert the format to the MS version from the Oracle version.  You are adding two steps, convert and verify (or convert and hope it works if you don't verify).

Ever seen something translated out of English and then back again?  How many changes were there from the original?  I read just enough French to see that quite often the French and English versions of things are often subtly different.

How data is stored is pretty basic ( Codd ), but you don't store it in an XML format. That would inefficient. Instead if you wish to move that data from one database to another, you will need a common specification to communicate that data to another system that both systems understand. XML fills that need in the same way the HTML specification(s) allows all the different browsers and OSs to render the same information and data on your screen. Problem solved. Most of the information in the XML doc is about how the data relates and what are the entities.

The government can choose whatever 'format' (poor choice of description) it wants, but I think it is counter-productive to then impose that standard on others when a solution exists already to communicate the data and its schema. One of my concerncs is that because government moves so slowly when it comes to standards that imposing a regulation like this will hobble improvements in data storgae technologies.

[Bonk]

I don't believe that Nemesis is suggesting that the government arbitrarily impose standard formats on all, but rather use open standard formats internally, to encourage competition, avoid vendor lock-in, preserve data retrievability in the long term... etc. etc. It is just good sense. Besides, having a government totally dependent on MS and Oracle just does not seem right.

Now as for formats, XML is not really a file format in my opinion, XML is a data structure definition contained in TEXT files. That is OK with me, as I'll always be able to open them and examine the contents. (that does not mean I approve of the concept of XML, and the javascript that usually goes along with it).

CSV (like XML) text files are also interoperable with all major database engines.

So, if we want to talk file formats, what is important is binary headers, bit depth, byte order, character sets, etc...

But the original point is that the Canadian government might be starting to take open source software seriously. I'm happy about that.   

[toasty0]

I don't believe that Nemesis is suggesting that the government arbitrarily impose standard formats on all, but rather use open standard formats internally, to encourage competition, avoid vendor lock-in, preserve data retrievability in the long term... etc. etc. It is just good sense. Besides, having a government totally dependent on MS and Oracle just does not seem right.

XML ends dependence on any one vendor.

Now as for formats, XML is not really a file format in my opinion, XML is a data structure definition contained in TEXT files. That is OK with me, as I'll always be able to open them and examine the contents. (that does not mean I approve of the concept of XML, and the javascript that usually goes along with it).

CSV (like XML) text files are also interoperable with all major database engines.

If I called XML a file format I mispoke. It is a specification that happens to use a text file.

CSV has drawbacks that XML solves and is one of my reasons for being a fan of XML over CSV.

So, if we want to talk file formats, what is important is binary headers, bit depth, byte order, character sets, etc...

But the original point is that the Canadian government might be starting to take open source software seriously. I'm happy about that. 

A good thing indeed a long as it is Open Source and not just another attempt by government to legislate technology.

[Nemesis]

I don't believe that Nemesis is suggesting that the government arbitrarily impose standard formats on all, but rather use open standard formats internally, to encourage competition, avoid vendor lock-in, preserve data retrievability in the long term... etc. etc. It is just good sense. Besides, having a government totally dependent on MS and Oracle just does not seem right.

No vendor lock in is the key.  Not just for the government but for those who have to deal with them.  When I go to a government website I should not have to use the same browser they use.  When I send them a data file I should not need to create it with the  same one they use to read it.  And so forth. 

[Bonk]

http://www.x-hive.com/products/db/specs.html

Snicker... giggle... guffaw... 

And he wants money for it! That's the best part! Hoooo... funny stuff.

[Nemesis]

Link to full article

The French Gendarmerie's gradual migration to a complete open source desktop and web applications has saved millions of euro, says Lieutenant-Colonel Xavier Guimard. "This year the IT budget will be reduced by 70 percent. This will not affect our IT systems."

The Gendarmerie's decided on a strict open standards IT policy in 2002. The 105.000 heads strong police force at that time used an out-dated IT system which was expensive to maintain and not able to interoperate with other police forces or custom agencies. Strictly maintaining of this policy for instance lead SAP to adapt its human resource application so it could be used with a web browser. "This means we can move to a competitor's human resource application at any time.

The power of a government, when a proprietary company wants the business they adapt to what the government wants to buy.  Also with open standards the ability to interoperate is automatic providing the government as a whole adopts a single standard.

Guimard says the Gendarmerie since 2002 found out that open source applications usually are better at handling open standards than proprietary software. Moving to centralised Imap servers for email, lead the organisation to deploy Mozilla Thunderbird on the desktops. Making all the web applications work equally well,resulted in the roll-out of Firefox across the organisation. "Users need no training to use a web browser."

That no training element must also save on costs for them.

[toasty0]

"Users need no training to use a web browser."

Is this a function of open standards, or good hiring practices?

[Bonk]

Oh man, that is too easy... something about donuts perhaps? (or petit-fours in this case?)

The browser as a platform is a bad way to go if you ask me. For lots of reasons.

Make them all learn arcane command line interfaces (a good OSS tradition) like we had to back in the good old days. 

[toasty0]

Yes. Real mean eat lunch at the cmd line!

But the rich bastiches win the game with a GUI