Maybe this is already common knowledge, but if not it might be helpful:
There's a big database "GOV - the genealogical gazeteer" covering Germany and quite a bit of the rest of Europe here:

http://gov.genealogy.net/Locale.do?language=en&country=us

I think it's been designed and is maintained by the people behind

http://www.genealogy.net/

This is mostly a German project, a vast resource for everything concerning genealogy. There are a lot of intelligent and knowledgable people involved there -- at least that's what I'm assuming ;-), being just a user of their website(s) and services. But maybe you could get some of these people to contribute here? I'm sure they could add valuable perspectives.


Contact adresses I've found on the website are:
gov-support@genealogy.net
vorstand@compgen.de

Mailinglists of potential interest can be found at:
http://list.genealogy.net/mm/listinfo/gedcom-l
http://list.genealogy.net/mm/listinfo/genealogie-programme

...to add to my post:
The GEDCOM 5.5EL (meaning "Extended Locations") proposal was developed by the "Society For Computer Genealogy" and a number of authors of German geneaology software.

I would like to suggest that this forum should separate the discussion of the data formats for locations from that of the universal gazetteer, the data base linking place names (spatial referencing by geographical identifiers) to coordinates. I am sceptical that the universal gazetteer is actually feasible, given how poorly mapped large parts of the world are, the political issues over authority, etc.

The problems you have been discussing of time-dependent spatial referencing (by geographical identifiers and/or coordinates) are not unique to genealogy, of course. Having been involved in ISO/TC 211, Geographic information/Geomatics, for over a decade I am somewhat biased, but I think that our suite of standards (including ISO 19136:2007, Geographic information - Geography Markup Language) will cater for a lot of your needs, and anything that is missing can be added to the standards.

While ISO standards have to be bought, unfortunately, there is a comprehensive standards guide that can be downloaded from the ISO/TC 211 web site at: http://www.isotc211.org/

Regards
Antony

I think we're veering off here a little bit, at least from my perspective.

I do NOT advocate that place entries within BetterGEDCOM be tied to ANY external or universal geographic mapping system or anything like that. People should absolutely be able to specify places as they want to, whether anyone else understands them or not.
I am merely suggesting we add support for passing information about where a particular geographic mapping system has reference to the place the original user is referring to. The entirety of the mapping of that place, conformance of recording that information, etc., should and would be something done within the genealogical application the person was using.

Remember BetterGEDCOM (for purposes of this initial project) is merely a way to store data uniformly between programs. We're not trying to add features to actual genealogy software but we do want to accommodate pass along all data that has been accumulated. Right now there are lots of ways users map the places in their genealogy databases, and all I'm advocating is a way for BetterGEDCOM to be able to receive that mapping information and pass it along.

Regarding URIs, I am not sure this is needed or appropriate, as that sort of thing is something the app developer would have to decide upon and implement. However, if URI reference format is appropriate, then great.

Dallan, I totally agree that inability to edit discussion comments or even format them is extremely annoying, particularly for a wiki...

Dallan said, "The problem with using user-entered data as the basis for your place database is that users are not good at entering place data..."

Yes, I know this, and this was actually debated extensively during one of the GEDCOM 5.x revisions 15 or 20 years ago. I've read some of the notes. On the one hand, it's a nice concept that everyone would have nicely identified places that conform to some wonderful, universal, noncontroversial geographical mapping system somewhere. But in practice, things just don't work that way. Even if such a database existed, genealogists would still want to be able to cite inexact, vague or idiosyncratic places. I fully support giving folks the option to pass this sort of geographical mapping equivalency information but I completely oppose making such a system mandatory. This would be like insisting on everyone map their relatives to people who have been recorded in census records. It sounds vaguely ok but in practice it's a horrible idea.

testuser42,

WOW - This database is incredible. Germany is the most complicated country I can think of with respect to historical places, and these guys are doing a terrific job. THANK YOU for sharing it.

greglamberson,

I'm not arguing that every user-entered place must correspond to a standardized place. Just that "automatic gazetteer generation" (google it sometime) using user-entered places as your source has problems associated with it. If a records-manager can map a user-entered place to a standardized place, then the user-entered place can be shown on a map without the user having to enter lat/lon themselves; for user-entered places that can't be mapped you don't get this feature. That's all.

Dallan, Perhaps this explanation will help resolve our different ideas on location identity and management.

BetterGEDCOM, won't care how places are entered. BetterGEDCOM is a genealogical Switzerland: It is neutral on matters of preference for one system or another. Thus, however location information was entered, BetterGEDCOM doesn't care. If there were some location qualifier that the software program used, largely guiding users to enter places that were identifiable, then great. BetterGEDCOM doesn't care. However, the information that identified that place in that system should be able to be exported with the location value and subsequently imported into another software program. Period.

To be useful, would software programmers have to use this feature? Absolutely. Are they required to? Absolutely not. Is it within BetterGEDCOM's scope to require software developers adopt some location management system? Not within this initial project.

Here's an interesting idea: Could we develop a BetterGEDCOM Location Extension that did specify the sort of system you refer to? Absolutely. Such a project would be exactly the sort of thing that BetterGEDCOM would like to see develop as a secondary project. Within the scope of this initial project, however, such an effort would be counterproductive in that it would force developers to adopt a particular approach to location management, and that would be a deal-breaker for lots of vendors.

Does this resolve our differences?

Sorry, I didn't think we had any differences. I'm fine with storing places as free-form text strings.

xvdessel:

I have expressed it in the "Location Entity Over Time" thread: http://bettergedcom.wikispaces.com/message/view/Location+entity/30668879

but I'd like to re-emphasize that I'm totally against a location-time stamp combination.

I feel a specific location should be a single entity. If it changes names over time, then that should be within the entity. My example (using extended GEDCOM) is:

0 @P43@ PLAC
1 NAME Townsville
2 DATE From 1832 to 1912
1 NAME Citiesville
2 DATE From 1912
1 LATI N18.150944
1 LONG E168.150944

I believe this will quite easily handle DearMYRTLE's concern.

In other words, the entity should be "Location" and NOT "Time-Location".

Adding time onto any entity (known as time-stamping) gets into huge complexities. If identical places in different times are kept separate, then how will it be known they are identical? You'll have to add a connector - the first level of complexity. I was involved in an SAP data warehouse project, where the designers insisted on time-stamping the data. It was a disaster.

If a person changes name, or sex, or hair color, do you want to make them a "time-person" entity?

I also am one to believe that events can and should be assigned to locations, rather than just people, when it is appropraite in that they pertain to the location. (e.g. Fire in 1848), but that's another matter.

... and I don't like places as free-form strings.

People are messy and uncontrolled, and without structure, they will create a mess of places. That will make it impossible for a program to put together a useful Place Index. Putting data together from multiple people and trying to organize places will be a disaster.

I'm for structure here.

Greg:

I would not start a secondary project before the first one's even off the ground.

If a secondary project is required, then the model selected is much too complicated.

Even talking about a gazetteer of places is way beyond what BetterGEDCOM is about.

Louis,

Adding a time element to a location entity is not at all the same thing as time-stamping. Time-stamping refers to adding an indicator to the data specifically related to when the data was last changed.

If the key for the Location entity was a combination of a time-place, yes, that could be a problem. I think the key for nearly every entity we use will be a UUID. Issues of duplication you raise will therefore be moot.

Regarding location structure, that's the job of the software app. We're just trying to provide a place for data to reside, however it looks when we get it.

Regarding other projects, there are several future projects in mind. That has no bearing on this project.

With regards place structure -- what about a US town that existed historically in one of 3 counties in a state, but now exists in a 4th county?

What about working with GoldBug? Art has passed away, but his coder still works the AniMap product.

When I proposed the idea of a reference database, I think (and this seems to be comfirmed by several comments above) that we should go for a database that includes the time as a dimension: no location should be defined without an associated time delimitation (which could be infinity on both start and/or end time). Obviously, each location, no matter at what time it existed, should also have world coordinates (or an area definition if possible). And ideally such a location should be hierarchically linked to related larger locations (town,county, ..., country). It could be more than on higher location, if the parent location changed in some way during the lifecycle of the location itself (e.g. a town can exist for 300 years, but have belonged to 3 different counties over that period).

I would therefore propose the naming convention of a "time-location" as follows:
A time-location is a point or an area that corresponds to a commonly known location during a specific period of time. One time-location could have multiple area definitions if its area changed over the period of its existence. A time-location could be independent of any organization (e.g. nature locations), but most time-locations should be related to an organization structure (administrative, judicial, religious, ...) which often implies a hierarchical relation to one or more time-locations of a higher level in the same organization structure.

For users, this could mean the following: If a software supports such reference db system, then you would first select a number of time-locations that you want to use in your data. If you selected one time-location (e.g. your currently existing town), the software could recommend you to also select historical versions of that location, or related areas that cover a similar location (e.g. a parish).
Whenever you then want to enter a location for an event (e.g. a birth), you would be presented with locations that are time-relevant: It does not make sense that a birth in 1970 took place in a town that only existed up to 1920. Moreover, the system could insist that a baptism should be located in a parish rather than a town. But still, as both a baptism and a birth event have time-locations with deductible coordinates, a good software could match one to the other.

What is important for the end user here is that such system implies a more strict data entry. Just have a look at this page:
http://en.wikipedia.org/wiki/Alexandria_%28disambiguation%29
and you will understand that a software that stores simply "Alexandria" as a location can never expect to have such data exported and reused with any degree of usability. Hence, the end user should first indicate which "Alexandria" he commonly wants to refer to (e.g. Town=Alexandria, County=Jefferson County, State=New York, Country=United States of America) and for which time periods (depending on the historical changes related to this town). When exporting, the exact data (in a format to be decided, but this is less relevant to the user) can then be provided, which ensures that the receiver (software and hence the end user) cannot misunderstand that time-location.

To all previous posters:
are there any open initiatives today that target such a time-location database?

Xavier

I agree with Greg that we should not enforce the use of one location system or another (Google maps, ...), but the standard should reference in a unique way a time-location (see above) in some commonly available database. What I would therefore recommend to have a look at URI's. Most people only think of http as a URI scheme, but there are many others, both official (e.g. mailto for mail addresses, geo for physical locations) and unofficial (e.g. secondlife, for a location in the virtual world). We could go for such a structure, simply stipulating the structure of the URI. This does not bear any link to databases or software that stores/uses it (just like http: can be used by many web servers and browsers).

An example syntax could then be:

timeloc:<server>/<unique-time-location-id>
or
timeloc:

2 - If you export your data for someone else, do you want then to provide that complete reasoning, or only the conclusion data?

I would like to control what I export period.

I may or may not want to exchange / export either of those pieces of information.

If I were a professional genealogist, which I am NOT, I might want to export Only the Conclusions I reached.

If I were sharing with a family member or a newly found cousin, I probably wouldn't share either of your items. The person I am sharing with, may need to draw their own conclusions based on the material they have.

Control / Options to include or exclude is what I would want.

Hope that helps,

Russ

3 - If you import data, do you only see the imported data as a source (input to your conclusions), or could you be willing to import the complete reasoning as such, and allow it to match with the reasoning you made, e.g. to determine why the same source data leads to different conclusions in the head of someone else?

First, I would want to view that information OUTSIDE of my own file. Initially, I don't want it part of my file.

I would hope that my program would let me completely or selectively import and merge information from that file, into my file.

When I do select to include information, I to include a complete 'record'. Record may not be the right term.

Taking the "Event" as described in Question 2, I would want the attributes of that Event, included all of the Source-Citation information what was recorded in the file.

I may or may not want the conclusions or information supporting that conclusion. If I were to include that, I would NOT want it to over write what I might already have.

I might want to compare the two conclusions and draw a new conclusion, delete a conclusion, combine the conclusions.

Options is the thread here.

Thank you,

Russ

4 - Do you see export & import only to exchange data with peer researchers, or also to allow people to change from one software to another with a minimal loss of data? It should be clear that each of these uses has clearly other requirements!

I think that answer is yes.

I may or may not know if the person I am sharing my research with is a Peer or not.

Bottom line, Yes I want to be able to share my research with a minimal loss of information.

What happens to the date after it is received, I can't control. I can guess it will be completely accepted, completely rejected or somewhere in between.

I haven't merged a GEDCOM file into my main file for years. I have my "standards" in data entry.

That does not mean that I don't review a GEDCOM file that I receive. Usually what is helpful is the Source of the data in that other file.

Since we are talking about Importing and Exporting, I would further request to my genealogy software developer that each entry that I do include in my file, from that GEDCOM use, in addition to any and all Source / Citation information, include the information as to where that information came from. That is, the information that is in the very beginning of that GEDCOM file.

So, lets take an Event. That Event would have a Source-Citation associated with it. I want that included. But, in addition the Filename, Date of Import, and Originator of that File in a format that would be consistent with Evidence Explained!.

Please don't read that the Source-Citation for the Event in that file needs to meet that standard, but the originator of the file should be.

Does that help?

Russ

5 - After you imported data X into your data Y, when you export that for someone else, do you want to flag the X-data as yours, or should it indicate the original source where you got it from? If X contains data Z, do you want to see the complete path via which you obtained it?

Yes, see my answers to Question 4. This is a very good question.

I hadn't really read this question when I answered Question 4, but I can tell you, that every once in a while I find an event that does not have a Source-Citation associated with it, that I know that I received in an early GEDCOM file, which is why I asked for that new Source-Ciation (GEDCOM file origins).

I certainly would want to pass that information along to any export that I would produce. In other word, Here is where I found this information. An online source, a physical document, a GEDCOM file.

Russ

6 - If you store a location, do you want the system to insist on a limited list of values (e.g. all known towns, etc) as that could increase the reuse of your data later on (i.e. to cope with translation and other issues)?

I think I answered this question on the Location discussion.

The offering of a suggested Place Name / Location is helpful, but I may or may not accept it.

I currently accept an Place name, based on a current place name, but if I knew that at the Time of the Event that Place had another name, I would add an additional, historically based, time sensitive Place name.

The best example is the where I lived in the late 1950's. It has a different name now. So, the place name when I lived there, in the 50's is what was entered. But, I would also use the current name to take advantage of some other features that the software provides.

I should be saying this on each post. This is just one User's opinion.

Russ

Russ,

thanks for your input! I hope many others will follow.

Yes, I'm a genealogist (if I have time for it) and 15+ year professional experience in IT & software (banking sector).

I'm living and researching in Europe where very often (period <1750) the information in the archives is very limited (of a baptism only mentions father & mother, godfather & godmother. If you are lucky the priest mentioned in which church the parents were married, but almost never any dates are mentioned). Hence, the deduction mechanism is of key importance. However, I don't know any software that allows to record first 20 separate events, and then, one by one, allows to stitch them together into one single conclusion. Typically, the fact of NOT finding something is often also a piece of data (e.g. if a individual has only 1 birth in the region that matches its name & approx. birth date, the fact that the neighboring towns don't have that name/date combination is an element to decide which birth act to accept).

By my IT background, I learned the importance of a good model. But I also understand that end users view things from their perspective, and it is finally for them we do the work (including for ourselves when we use software).

And from your answers, it may be clear that your requirements seem to be quite high, which is logical. Whether all that is achievable is another story! But others have already commented on that (E.g. will software vendors use a new standard?).

Xavier

Xavier,

Thank you for your reply.

I spent 30 years in the communications industry with that last 10 or so, helping to write User Requirements for several software systems.

I have spent the last 10 years working / helping users of a specific Genealogy software program. Some of my requirements, I hope, reflect some of User Requirements that I have seen in the past.

I do hope that other Users jump in and help with these requirements.

If I don't start high, I won't get there.

I also hope that some more software vendors participate as well. After all, this project's hope, is to allow Users, like myself, Share my research with others.

But, I am happy to participate in the development of the requirements.

Thank you,

Russ

Personally, I think the question is asking the wrong question. But this is purely from my perspective as a software consultant who has designed systems for clients with varying levels of technical knowledge.

I think the goal should be to involve the participants in a discussion similar to Xavier's list of questions above.

In order to know how to design the system you have to know what your goal and expectations are. You have to know what features you want, what data you need to collect and any actions that need to be performed.

The technical people should help by guiding the questions in constructive ways so that answers to questions help to construct the model. If the questions are posed correctly, the data model builds itself.

For example, I think it is safe to say we know some base objects we need to collect information on. I think it's also safe to say we know some base requirements we have for any data model.

So a good starting point would be to define a base model from which we need to add.

For example, we know have people.
People have events, places, references, etc. associated with them.

We also know we have requirements about how to handle the data. Like the excellent questions and answers provided above.

I'd like to add, that I think a great deal of the work and a great deal of data structure could be gleaned from the Gentech data model. Not that I'm advocating their entire solution, but the data elements are a good source of information. I probably wouldn't implement them the way they went and I skimmed over a great deal of the cruft before the data definitions. Plus it would be overkill for some people, like me. But then, there's things there that I'd use and there's things in Gramps I don't. (A preview button on this page would be nice.)

Brian,

The requirements are "simple" from this Users point of view.

I want to Share my family history information, in my genealogy software, with another User, who may or may not use the same software. No data to be lost, and there may be Media within my file.

The project is to lay the ground work for the Transport of that data.

The request is not who to present that data, only get it from one place to another.

I understand that is not an easy task.

One End User's opinion.

Russ

Hello Xavier,

thanks for these very good questions. I agree with a lot or nearly everything that Russ wrote.
So maybe I'm just putting the same thing in different words, but here it goes:


- do you want your computer software to insist on storing the complete decision process you made to reach a conclusion in your genealogy?

I want to be able to store the decision process that made me reach the current conclusion. A simple note would suffice, I think.
If I come to a different conclusion later (because of new evidence or new understanding of the old one), I want to save this new conclusion and the process. But I want an option to keep the earlier conclusion stored away as a reminder (with a time-stamp?), so I don't make the same mistake again. If I decide to trash the old conclusions, that should be possible.


- If you export your data for someone else, do you want then to provide that complete reasoning, or only the conclusion data?

Everybody should be able to decide what he wants to export. Privacy might play a role etc.
But to be really helpful, I would export everything (sources/evidence, reasoning, conclusion) about the persons and events that I export. I think the software should default to this behavior, and offer options to leave out parts.


- If you import data, do you only see the imported data as a source (input to your conclusions), or could you be willing to import the complete reasoning as such, and allow it to match with the reasoning you made, e.g. to determine why the same source data leads to different conclusions in the head of someone else?

I would be willing to look at other people's reasoning and conclusions. If I agree, I'll take their conclusions (maybe add a "me, too"). If I don't, I'll take their sources only (if I think they're useful to me).
If I only had conclusion data without reasoning and evidence, that would only be an aide to further research, if at all.


- Do you see export & import only to exchange data with peer researchers, or also to allow people to change from one software to another with a minimal loss of data? It should be clear that each of these uses has clearly other requirements!

Both.


- After you imported data X into your data Y, when you export that for someone else, do you want to flag the X-data as yours, or should it indicate the original source where you got it from? If X contains data Z, do you want to see the complete path via which you obtained it?

The complete path would be ideal.
If I didn't change anything about a source/evidence record, then it might be OK to leave the previous "owner"-info.
If I agree to a conclusion and don't change it, adding my "me, too" might give this conclusion more credibility for future researchers.


- If you do an import from someone who already provided you some data before, do you expect your software to eliminate duplicates (i.e. data that did not change between the previous upload) and flag new data or updates for you to examine?

Yes, that would be very helpful.


- If you store a location, do you want the system to insist on a limited list of values (e.g. all known towns, etc) as that could increase the reuse of your data later on (i.e. to cope with translation and other issues)?

Ideally, the software would offer me a list of previously used places (pop-up or autocompletion as I type?).
Places would come in an evidence and a conclusion version, too. If a source just says "Newton", I can only put that down. If I believe that this Newton is the Newton in C County, I put that and my reasoning behind that into the conclusion place.
If I enter a new place without a source, it is just a conclusion place. I should be able to add a "certainty" to this, just like any other conclusion. Of course I should be able to add sources later.
The conclusion place should also collect data that isn't about any event of a person. Many things just concern and define the place, like a change of name, or government.
I would love a geneaology program that helps me gather more data about a place, like coordinates, historical and current information.
Places could be nested in other Places, like Country - State - County - City - Neighborhood etc. Since these relations are only valid at a certain time, it should be possible to save a time-span with these nestings, and to allow multiple "parent"-places.


Again, thanks for these clever questions. I hope my user's POV may be of a little help to you techie experts who do the hard work :)

Russ,

It's very apparent that your requirements are anything but simple. But then you might also think that walking on two legs is simple. In reality, very little is simple. First off you want to use "your" genealogy software. Which means any format or tool would have to be able to read the data from your genealogy software. This would possibly involve thousands of hours of reverse engineering of that program's storage mechanism. Or getting your genealogy software maker to write an export feature for the new format. But you have just given us a base specification from which to work.

Sadly, if we take that approach we're likely going to wind up with a model that is huge and cumbersome. As everyone will want to use whichever one they are already using. Which would encompass in the end every conceivable format.

But, I think you knew that when you wrote it.

Does your current software record everything the way you like it? Is there nothing missing? Is there nothing that you don't like about it?

Are we talking about building a data model and a program? It would seem to me, one result would be at least some useful code. I've built data models before and the code almost writes itself in the process, if done right.

My ideal is always to start simple.

We need a data model to have any idea where we want to go. This can really only be done element by element. However, it doesn't mean we can't use prefabricated structures and: cut out, copy and paste in the changes we want.

Take person for example, we seem to be of two minds here, one is an "evidence person" and another a conclusion person. Well let's simplify. You have instead a person. Jus a plain old person. If some evidence later points out that that person is two people, then all the model need handle is allowing a "split" transaction. You can take any piece of evidence in a person and split it out as another person with the same name. When splitting a person into two, you can choose which bits of data go with that person. Problem solved, and there is no need for a evidence person or conclusion person. You collect evidence on one person and later conclude it is more than one person. Keep it simple. Or at least obvious. After all if you're researching John Thomas Smith you're goal is one specific John Thomas Smith, not two or three or four. Why start with four and later have to merge them?

Alternatively, any decent genealogy program should have no problem merging two or more John Thomas Smith's. So users would be free to take either approach. Although, I'd think merging is more work than splitting. Many people have a tendency to take the hardest road, given free reign, myself included.

Of course, the problem is not in the data model, but in the implementation. Of course, I can see the other viewpoint, I have in one town. of about 1000 people, six Johann George Schmidt's all living at the same time, with births spanning a number of years, and several wives with the same first and ruf names and all with children overlapping. One couple is mine. Another his parents.

So, I say start simple, we'll complicate it fast enough. No need ot make any judgments at this stage.
We need the following type of data: person, place, source, citation, conclusion, event. We could also include: relation and characteristics.

Brian,

Yes, my request, as ONE User of a software program, is Simple. Get the Data from my Software Package and transport it to another End User, and 'don't mess with the data' in the transport.

To Get there, is not simple.

Does my program do everything I want? Absolutely NOT. BUT, if I start by asking for software limitations, or making the requirement simple, I'll end up with something simple.

The hope is, that this BetterGEDCOM group will lay out want the Users Really want and the technical types (not me) can help lay the ground work for the developers to come to the table to see how or what the developers can do.

Just so that you know, when I see "evidence person" I stop reading. I have a Person. I am gathering and documenting "who" that person is, through Facts or Events, along with that documentation. That "person" WILL change, based on new evidence. I WILL find conflicting information.

That said, I think you and I agree on this.

Russ

I agree with Tom that we need to have at least an idea of the high level model we try to achieve. How can you talk about "data about events" or "data about a person" if it is unclear which direction you want in your model, what you mean by "an event" and "a person".

Therefore, the first key decisions will most likely be around the scope: does the model cover only conclusion data with reference to sources (like GEDCOM does) or do we want to extend it to source data elements, the decision process, and the resulting conclusions.

Tom,
xvdessel,

I sort-a understand what data models are, don't want to guess at what that means, but I am not sure we are there yet. Clearly, developers need to know or create the data models.

This wiki was created, because we, Users of various genealogy applications have an issue. WE Want to Share our Research with others without loosing information and what is shared is selectable by the user.

Now, if I had a Data Dictionary for the software that I might be using, that is Field Name, its properties, and what it's used for, I might be able to come up with a data model. But, I don't want to do that.

Please help us create that Data Model. Oh, I think, to do that, we need to have developers at the table. That is what we are trying to do.

I am not sure that I care if the end result is a GEDCOM or some other 'thing'. All I know is that the current GEDCOM feature doesn't work the way I want it to.

Please help us.

Thank you,

Russ

Russ,

I can agree that an end user really does not care what and how the software does to exchange the data from one user to another. That is indeed an issue for the techies.

But that does not remove the burden from the end users, to come up with "business requirements". Let me put forward some questions to clarify what I mean by this:

- do you want your computer software to insist on storing the complete decision process you made to reach a conclusion in your genealogy?
- If you export your data for someone else, do you want then to provide that complete reasoning, or only the conclusion data?
- If you import data, do you only see the imported data as a source (input to your conclusions), or could you be willing to import the complete reasoning as such, and allow it to match with the reasoning you made, e.g. to determine why the same source data leads to different conclusions in the head of someone else?
- Do you see export & import only to exchange data with peer researchers, or also to allow people to change from one software to another with a minimal loss of data? It should be clear that each of these uses has clearly other requirements!
- After you imported data X into your data Y, when you export that for someone else, do you want to flag the X-data as yours, or should it indicate the original source where you got it from? If X contains data Z, do you want to see the complete path via which you obtained it?
- If you do an import from someone who already provided you some data before, do you expect your software to eliminate duplicates (i.e. data that did not change between the previous upload) and flag new data or updates for you to examine?
- If you store a location, do you want the system to insist on a limited list of values (e.g. all known towns, etc) as that could increase the reuse of your data later on (i.e. to cope with translation and other issues)?
Each of these questions relates to possible options or key choices in the data model. E.g. the last one could mean that a data model stores a globalized identifier of a location rather than a free text location, but more importantly that locations relate to a centrally managed reference database which could make it language-independent etc.

I hope this clarifies a bit why the data model is important.

Xavier

Xavier,

I sorry that you concluded that a Data Model was not important. I KNOW it is.

What I will try to do, is to answer your great questions, one at a time, over the next couple of hours. I want to think a couple of them through.

I do have a question for you though. Do you do family research and do you share your research? I ask, because I will then know how to answer the questions that you asked.

Thank you,

Russ

-- Oh, this is the dialog we need here. Thank you.

Xavier,

1 - do you want your computer software to insist on storing the complete decision process you made to reach a conclusion in your genealogy?

Wow, don't know how to easily answer that question.

The software show 'save' information as I leave a field.

Now sure what a 'complete decision' means to you.

I enter information into fields into the program. Most of the fields are Events. Today, an event might have four attributes or fields. And this is just an example:

Event / Fact Name:
Date of Event:
Place of Event:
Description of the Event:

I may or may not have the three fields completely filled in. I may only have the Event and the Date of the Event.

Doing research, I want to document that Event. Where did I get that information from?

I enter what I have, and cite the source of that information.

No conclusion yet, just recording the event.

I may have the same Event / Fact from various sources, each providing pieces of information.

Each would be recorded separately. If I find the same information, previously recorded, in another source, I'll add that Source-Citation to the existing event.

At some point in time, I need to Evaluate what I have collected for that Event. Following that Evaluation is when I might reach a conclusion.

I may conclude that I can not reach a final conclusion, as I might be missing some element or I might have to resolve some conflicting information.

That is why I am not able to completely answer your question. But, maybe I did.

Thank you,

Russ

Here are a few more things that would help improve existing gedcom support:

(1) A standard way (like a _living tag) to mark someone as living. Currently when someone has no dates, in order to determine whether they are living (so we don't publish data about them publicly) we have to start looking at dates on their relatives. It's a pain.

(2) Standardize on name pieces, or at least a requiring full-names to be in "name-prefix given-name(s) /surname/ name-postfix" format.

(3) For source citations, standardize on either a text sub-field or a note. Currently both are allowed by the standard. Some programs use both, some use just the text and throw away the note, others use just the note and throw away the text. It makes it nearly impossible to export a gedcom that will work for all programs.

(4) Standardize on a way to represent the type of parent-child relationship: adoption, biological, etc.

(5) You may want to allow people to attach sources and notes to relationships (e.g., child-parent, or person-spouse), in addition to sources on individuals and family objects.

"
(4) Standardize on a way to represent the type of parent-child relationship: adoption, biological, etc."

At the risk of setting off a firestorm...

I've said this before and I'll say it again... there is only one type of genealogical parent-child relationship-biological. any other type such as adoptions, fosterings, et al are mere social relationships and *not* genealogy and as such have no place in genealogy reports such as genealogy charts, reports, etc.- they are family history; genealogy and family history are *not* the same thing.

Andy,

To avoid your expected firestorm, I think that I can say that many researchers don't agree with you, for at least following reasons:
- many children are born from another biological father that the one who pretends (or even thinks) he is the biological one. This was true in the past, and still is (proven by research). Hence, unless you have very extensive DNA test budgets, forget to reach back 300 years in a purely biological way.
- What people become is a combination of nature (biological relations) and nurture. Many researchers value the latter, even if the former may be more important.
- Even biologically speaking, there is an issue, if you take into account surrogacy. A child can be born out of carrier mother X under a surrogacy agreement, grow up within family of parents Y and Z, but biologically inherit the genes from U and V. And wait till science can combine exotic pieces of DNA from different people to build tailor-made babies.

Hence, I think point 4 still stands as a point for most of us.

Xavier

Andy,

Thank you for the firestorm topic. I do understand what you are saying.

The thought of a BetterGEDCOM was for Sharing of Information. So, it's a way to transport information in my Genealogy software with you with your software.

The issue of Family History vs Genealogy is interesting. I would throw in another Group to the mix, 'Name collectors'. Please, we don't need to take a tangent on that firestorm.

The difference, I think is not the Transport of information, nor the software that is used. Be that online on a computer you and I might us. It would be how You and I use our Software.

So, for the purpose of the BetterGEDCOM project, you and I need control over what we see or bring into our software. In other words, WE need options on what we USE. The transport needs to be transparent to the use of the data that is included in the transport.

If I am close in the three categories above, we need to be able to share what we have.

Let me throw "One Name Projects" into the Name collector category, just for discussion. I know of a One Name Project that I would like to see information from. The BetterGEDCOM project is implemented (at some point in time), I would want that One Name Project file sent to me. I would open that file in my software. During Import, I would want to control what is brought into my new file on my computer.

Lets say, that in the file, during import, I find the ancestor I have been looking for. I'd like to be able to Identify that specific person, Bring in ALL of his ancestors, siblings, spouse(s), and descendants and toss the rest. That is what I want to look at. I may or may not bring that information into my own file. But that would mean that I have ALL of the information, most specifically research material (Source-Citations), and not loose anything in the file that was created by the One Name project.

One step further, in reviewing that One Name project file, I see another Ancestor, who I know about, but is in a different family line. So, I would want to go back into that One Name project file, and export another set of Individuals, into a new / different file.

I am not a Name collect, but not the Genealogist that you are. I have become more of the Family Historian that you mentions.

Won't address the Name collector group, but for you, and I do hope you reply to this, that You be able to control ONLY the Genealogical Parent-Child relationships that you want to see. If we, the folks helping define our requirements, we should be able to "toss" those social relationships that you mention and not have them included when imported. So, the marking of relationships are very important.

I would just add, that I would want to see a list of those 'tossed' relationships at the completion of the opening of that file.

All that to say is that we need to have the flexibility to transport data collected by the End User.

One User's opinion.

Russ

Russ,

What I am saying is that yes, we do need a way to show social relationships; but that way should *not* be by tricking the software into believing that that relationship is in any way genealogical in nature. In other words, a child adopted by someone should *never* be shown as a child of the person who adopted them.

Presently there is no way to do this except with notes unless one links the child as an actual child of the adopted parent.

We need other linkages that clearly define these social relationships that will *always* show these distinctions, be it in charts, family group sheets, narratives, or whatever so that a step, foster, or adoptive relationship can never- under *any* ciscumstances, be mistaken for an actual biological relationship.

That is what I'm asking for.

Andy,

Do me, that is a software requirement, not the vehicle to transport the data. The transport has to be transparent, not to make any changes to what is presented.

It's in the export or import of the data, in my humble opinion, is where your requirement goes. By the way, I am supporting what you are asking, just don't want the BetterGEDCOM to do that.

Russ

Andy,

then I can clearly follow your point. And indeed, we may need a wider area of relation types.

However, I don't think it is up to the standards for data exchange to define how users want to view and/or print the relationships that are maintained by the system. A godfather has also a relation, be it a different one from a parent. It should be left to the software to decide which options it gives to the end user about this. Some software will keep it simple, others may indeed use a specific attribute (line style, ...) to indicate the particular relationship or could even allow the user to filter non-biological relationships.

But we agree on the bottom line that the exchange standards need ways to define all types of relationships.

Just out of curiosity: In the case of a carrier mother who puts a baby on the world from an implanted embryo, who do you flag as biological mother? The carrier mother or the woman who gave her egg cells to build the embryo?
And let's take it a bit further: Like Dolly, one could have a fertilized egg, remove all chromosomes and replace them by the chromosomes of an adult person. Note that the mitochondrial DNA is probably still the one of the fertilized egg, not of the adult donor. Who would be the mother here? And the father? I know it is future, but when scientists did frog cloning, nobody believed it was possible for mammals.

Xavier

Xavier,

In my mind, it is always the "sperm Father" and the "egg Mother" of the fertilized egg who are the biological parents- no matter what my be done to or with that fertilized egg at a later date.

Andy - I wholly support the idea that the exact relationship of a child to a so-called parent should be capable of exact definition.

My only point would be that BG _must_ accommodate family historians not just genealogists. Over in the UK many (most?) of us consider ourselves family historians, looking at the wider aspects of the family in all its forms.

Adrian,

Limiting myself to your comment, "...BG _must_ accommodate family historians not just genealogists."

Personally, I use the term "genealogist" to encompass both concepts. The thought of one without the other is repulsive to me.

I have lately begin to think of genealogy as the collection of the data and family history as the analysis of that data, but other than that, I am unable to draw a distinction.

Rest assured, the needs of anyone considering themselves either a genealogist or a family historian are exactly what we should focus on. You'll get no disagreement there.

Re: Dallan's evolutionary GEDCOM extension suggestions:

I've put the 6 mentioned here, with comments. This should be an entire project. This may be more important and have a larger impact that the BG project itself.

(0) A standard way to represent their UUID. I think that this has evolved to be _UID.

We also need a way to represent a set of these. If two person's are later found to be the same person, then we need a way to represent that, and to show which is the primary UUID. We have thought some about this and how it can be used to later merge information back (say after two researchers have made edits).

(1) A standard way (like a _living tag) to mark someone as living. Currently when someone has no dates, in order to determine whether they are living (so we don't publish data about them publicly) we have to start looking at dates on their relatives. It's a pain.

This is an interesting bit of information somewhere between evidence and conclusion. Gramps has a very sophisticated program to compute living (also used for computing estimated births and deaths, which can be added and marked as "calculated" dates). But a living tag is interesting (we use a living mark in our web-based app, because computing living status can be expensive). The living designation might need to have justification (linked to the person's date that provided the evidence, in case that changes). Also, it should have a current date attached to the living status guess, so that the date can be taken into account later. Perhaps add estimated DOB, and DOD, if missing.

(2) Standardize on name pieces, or at least a requiring full-names to be in "name-prefix given-name(s) /surname/ name-postfix" format.

Yes, these distinctions are lost when we go to GEDCOM. Gramps has one of the most sophisticated name representations of any application. Just recently we added multiple surname support. But most is lost in GEDCOM export.

(3) For source citations, standardize on either a text sub-field or a note. Currently both are allowed by the standard. Some programs use both, some use just the text and throw away the note, others use just the note and throw away the text. It makes it nearly impossible to export a gedcom that will work for all programs.

How could this be done?

(4) Standardize on a way to represent the type of parent-child relationship: adoption, biological, etc.

Ideas?

(5) You may want to allow people to attach sources and notes to relationships (e.g., child-parent, or person-spouse), in addition to sources on individuals and family objects.

Yes, another thing that Gramps XML handles, but is lost in GEDCOM. Ideas?

-Doug

@AdrianB38,

Modern genealogy recognizes extended families. See _Numbering your Genealogy_ for established standards (US) about recognizing adoptions, step-children, etc.

Now, how well modern genealogical software is able to support those standards, well, that is probably a whole other party.

Hope this helps. --GJ

Dallan,

We certainly want to plan for the future and encourage adoption of best practices, but we are most immediately concerned with the practical ability to import and export data. Here's the core of the immediate project's problems to solve as I imagine them right now:
1. Can we accommodate a conclusion-based data model and an assertion/theory model at the same time?
2. If these two cannot be accommodated elegantly within the same data model, where should the line be drawn between the need to capture data from existing products and elements which are desirable to adhere to good genealogical methodology regarding theories and building proofs for assertions?
3. What are the implications of the above on software developers who are tasked with making today's genealogical software programs map to this data model? Do we need to tackle less ambitious goals to accommodate practical development problems? Should we provide mere stubs for certain data classes (e.g., theory, assertion stubs for evidence/source data class) to signal to developers a path for future development without burdening them with wholly new data constructs to accommodate?

Regarding multimedia formats that could be compressed, this concept which GRAMPS uses has already been suggested and adopted in discussions elsewhere today.

The multimedia transfer problem has been discussed in at least three other topics, see the discussions Goal2 under Goals, and 'XML for Multimedia ...' under Home - and there is a third one I can't find.

I was just using that as an example of something that you could propose that would have clear benefits to users without creating a lot of extra work for the records-manager vendors, and therefore be more compelling for the records-manager companies to adopt.

The challenge that you face here is that I don't see any of the major records managers are participating. Without their buy-in, this effort will suffer the same fate as previous efforts have in this area over the past 10 years.

You need to convince at least some of the major records-manager vendors that your changes will be worth spending their development resources to implement. I think your most important goal is:

What changes can we make to the existing gedcom model that will provide clear benefits to users with a minimal amount of effort implementation effort required, so that our changes have a chance of being adopted by the records-manager vendors someday?

Until you address at that goal, this is just an academic exercise.

Let me give you an example of what you're up against with this effort to create a better gedcom.

Attaching a globally-unique ID to every person in a GEDCOM is a good idea. It means that when I share my GEDCOM with a relative or with an online service like new FamilySearch or WeRelate, and then I want to send an updated version of that GEDCOM later, it's easy for the online service or the records-manager used by the relative to determine which people have been updated, deleted, or added, because everyone in my GEDCOM is identified by a globally-unique identifier. This makes merging the changes much easier and more accurate. So we have clear user benefit.

The code to create a globally-unique ID is not difficult to write. It's less than a page. FamilySearch has even made example code available. So we have "easy to implement".

This idea has been around for a long time and several records-manager vendors have adopted it. By convention, they store the ID in a (non-standard) gedcom tag called _UID. So we don't face the problem of: "If I do it and nobody else does, there's no benefit". Others are already doing it, so there's clear motivation to the remaining records managers to add it.

Given the above, what percent of GEDCOM's do I see with _UID tags? You'd think it would be pretty high. It's less than 40%.

Furthermore, some of the GEDCOM's with _UID tags don't actually store globally-unique IDs in them. There are some records managers that allow the GEDCOM user to store whatever text they want to in the tag, and it's not required to be unique even within the GEDCOM, much less globally. So in order to take advantage of the _UID tag, I have to first determine if this is from a vendor that stores true globall-unique IDs in that tag, or one that allows users to store whatever they want in that tag.

Just getting vendors to store globally-unique IDs in a _UID tag in the existing GEDCOM file format would be a big win.

It's not as much fun to identify the current user-facing problems with GEDCOM, figure out how to fix them and work with the vendors to adopt your changes, as it is to come up with an entirely new file format. But if you want to have an impact, that's where you need to begin.

I couldn't agree with DallanQ more.

Perhaps a useful role that people here could play is creating a smooth path from current GEDCOM to BG, whatever it may be. For example, if we could all make sure that our current GEDCOMs supported an agreed-upon set of extensions (like _UID) then that would be a big step in moving forward.

Gramps is set to add a _UID (and related infrastructure) to its system. Any advice is welcomed. Anything else that we should all advocate to make common extensions?

Maybe (but if I remember well, this idea was already mentioned years ago in some of the historical GEDCOM-related efforts, and I even think I commented on that back then) we should build a leveled standard whereby a software can claim to respect the standard up to a given level.
We could say that level 1 is the current GEDCOM level, with only basic data covered. Level 2 could then be a more strict GEDCOM-like adherence, e.g. including things like Unique IDs, more exotic types of events, etc.
Level 3 would then be a new BG-level, but still at the level of conclusion data. It could include new concepts related to locations, date formats, event types and roles, character sets, gay marriages, Surrogate mothers, ...
Level 4 would then enter the raw event data and decision process.

To validate software, a set of files could be published for each level, whereby the test involves importing the test data into the software, perform a predefined data search (when and where was person X born?) and data entry (enter a new child for parents X and Y) and export all (or part) of the data again into the required format. After that, a publicly available program can compare the result file to a sample output file and give a percentage score.

I believe similar mechanisms are used to benchmark browsers and many other software.

Xavier

Dallan,

We have significant support from several major vendors. I wouldn't expect to see their participation actively, because they're commercial entities, and frankly, they won't. Who pays software developers to take part in completely open community efforts? Who allows their developers to announce their participation in an effort such as this? But they're certainly watching, and we certainly are considering that aspect of things very carefully. Don't worry about that. We're on it. However, you're not going to see some big logo that says, "SuperGenealogySoftware Pro 2010 Approved" here.

Regarding your UUID example, this I think reflects the fact that a great deal of users use old software that is simple and works but may not be under current development. This phenomenon is significant but it is also not of great concern. GEDCOM is very useful to many people, and while many of us think it's horrible, it actually works just fine for a great number of people. Every program can import GEDCOM, and why would existing applications simply abandon an import/export enigine that works and is already developed. It doesn't concern me that GEDCOM may still be in use in 5 or 10 years by the troglodytes. Heck, they're using software that's essentially 15 years old now! (And we all know who they are.) However, this phenomenon also won't hamper adoption by vendors with current products that are interested in remaining competitive in the commercial space where value-add is imperative.

xvdessel,

We certainly intend to have different standards that deal with different aspects of genealogical technology standards. We will thus definitely have different levels of comliance that adhere to different standards modules, as it were.

I found the reference to the FamilySearch provided code:

http://www.mail-archive.com/ldsoss@lists.ldsoss.org/msg00695.html

I can't believe it's been 4 years already. How time flies :-)

Java and other programming languages also have built-in functions for generating UUIDs.

Chris,

Gramps has census XML format that it has been developing, but only for the actual census columns. The data is stored in the Gramps database on events connected to each person. It would be very easy to extract the census data from the people and create an XML with headers, though. In fact we have a report that does that, except goes to text rather than XML.

For more information see:

http://gramps-project.org/wiki/index.php?title=Census_Addons
http://gramps-addons.svn.sourceforge.net/viewvc/gramps-addons/trunk/contrib/Census/
http://gramps-addons.svn.sourceforge.net/viewvc/gramps-addons/trunk/contrib/Census/census.xml?revision=461&view=markup

Hope that helps,
-Doug

Thank you. I'll have a look and see if I can make any sense of it.

Chris

My feeling is that the consensus drifted around the family entity being just a sub-type of GROUP. GROUP is needed (by some people) for all sorts of things like business partnerships and can also serve as the entity type for families. Presumably with some sort of sub-type to identify them.

Alongside this, my view of the consensus is that things like birth / adoption events involved the child and parents / step / adoptive parents, each with a defined role in the event and this gave you the exact relationship, instead of relying on this woolly "Child in a Family" concept which ignored that one could be a step-child of one parent and a biological child of the other.

BUT - since these were my personal preferences, I could be misinterpreting the consensus.

Mike,

My question would be "how do you handle relationships"?

What is your model's definition of a Family?

If you have a "family of 3", Mother, Father, Child, how are these handled beyond the PID?

Russ

@ Adrian,

you bring up a very good point about the old gedcom indi and fam area was mainly for generic default displays for navigation and placeholders of just a handful of records.
Over the years many users were forced to reuse those areas for other purposes like, adoption, business, bonds between to people, and step children. Many created sub classses inside this area making more complex than it has to be.

My FID (parents of a family) and PID (person individuals) functions the same way INDI and FAM records. BUT
PID holds no records
FAM only stors bonds between two individuals, no children listed.

here is a basic FAM.xml for just cause say the node 'F' for father 'M' for mother, other data like bond begin and bond end with places.
This is data selected by the user as default and is not a record but a place-marker for quick view. say 'g' fr group or family or bond what ever people view it as. displayed is seven groups.

<FIDxml>
<data>
<set><g>1</g><f>123</f><m>24</m><<(all the users generic display data)>></set>
<set><g>2</g><f>123</f><m>22</m><<(all the users generic display data)>></set>
<set><g>3</g><f>124</f><m>87</m><<(all the users generic display data)>></set>
<set><g>4</g><f>67</f><m>54</m><<(all the users generic display data)>></set>
<set><g>5</g><f>87</f><m>102</m><<(all the users generic display data)>></set>
<set><g>6</g><f>87</f><m>36</m><<(all the users generic display data)>></set>
<set><g>7</g><f>2</f><m>95</m><<(all the users generic display data)>></set>
<set>many many more entries here</set>
.........................
</data>
</FIDxml>

Now in the PID
<PIDxml>
<data>
<set><p>67</p><c>2</c><<(all the users generic display data)>></set>
<set>many many more entries here</set>
.........................
</data>
</PIDxml>

Ok Person 67 is a child of (family/group#2) from here you can navigate to thier mother#22 and father#123)

If you choose to select person#123, you will note he has a bond to other people women #22 and #24.

That is for navigating upwards throw records.
Downwards. Say the man#123 was choosen
the FAMxml pulls up two matches*

---

display*[G#1 matched with women #24]
filterPID match any for g=1
display*[G#2 matched with women #22]
filterPID match any for g=1

---

So whie looking at a family of a person, the siblings listed can display the nuclear family of 4 matching siblings AND a step child from a bond by marriage or blood from either spouse.

However adrain brings up a good point. adopted children with unknown genetic parent place-markers.
Note that the same way PID loops to the FID up and down via navigation. Captures all children and step children.

With Adoption EID.xml works the same way.
EID Marriage event can tag back to a PID as HUSB also the WIFE roll of the same event links back to the female. Same goes for a legal record Adoption.

Since these are not records but defualt user or research preferred display markers. The real thing is the EID.xml where you enter various class and type records. each eid line points back to PID's linking adoptions to a group, marriages between two person and witness to the same event. I will get into that shortly. I what to show the FID and PID grouping first and show the navigation does work. I did a functional test on my site and can display pedigree, household, and desendants very easily.

error correction
display*[G#2 matched with women #22]
filterPID match any for g=2

#22 was to be g=2 not 1.

Here for example my sister had two childern two seprate fathers
She is the pid in question # 161
http://www.stranskyfamilytree.net/gen%20project/view-family/family.asp?pid=161

Both children were put up for adoption and I have a way to do that. Adrain I know what you mean, this was one of my pet peeves also.

The EID.xml that captures the event "Adoption"
Each person plays a roll, birth mother, birth father if known. The child , the adopting parents each. One source record in the SID.
Each EID entry for rolls and data from source doc point to EACH of the corrosponding PID's

that is how you link many people to one event. flag eid lines sets as confirmed or disputed, or to review later. This is how you import export records individually, not mash them inside the INDI, pid have no recordds in them, PIDs are just visaul placemarkers for navigation, and a collected of records tied to a group piont per person.

LOL I am babbling again!!!