"datamining" thousands of IGC files

[Tibor]

Hi,

are there any open/free tools to datamine on database of many IGC files?

There is a threat that the free airspace for gliding in Slovakia is going to be significantly restricted. We need tools which help prepare arguments about airspace usage and frequency of our flights.

If I was starting from scratch I would probably try to import all the flights into a GIS database (PostGIS) and do the data cleaning and queries there. Does anybody have experience with this?

Thanks for you help,
Tibor Arpas

[Sean Fidler]

I have an intern currently working on a slightly different project for US flights in an effort to isolate for potential cloud flying incedents over thousands of competition flights. It has been very interesting so far. More later. He did create a batch method for adding large sets of flights (but only a few dozen at a time). Not sure what dbase he is using.

Send me your email and I will connect you.

On Monday, March 5, 2012 12:26:07 PM UTC-5, Tibor wrote:
> Hi,
>
> are there any open/free tools to datamine on database of many IGC files?
>
> There is a threat that the free airspace for gliding in Slovakia is going to be significantly restricted. We need tools which help prepare arguments about airspace usage and frequency of our flights.
>
> If I was starting from scratch I would probably try to import all the flights into a GIS database (PostGIS) and do the data cleaning and queries there. Does anybody have experience with this?
>
> Thanks for you help,
> Tibor Arpas

[Morgan[_2_]]

June 2010 Soaring Page 32

There is an article on Thermal Routes by Alfred Ultsch. I didn't see any contact info, but it might get you started.


On Monday, March 5, 2012 9:26:07 AM UTC-8, Tibor wrote:
> Hi,
>
> are there any open/free tools to datamine on database of many IGC files?
>
> There is a threat that the free airspace for gliding in Slovakia is going to be significantly restricted. We need tools which help prepare arguments about airspace usage and frequency of our flights.
>
> If I was starting from scratch I would probably try to import all the flights into a GIS database (PostGIS) and do the data cleaning and queries there. Does anybody have experience with this?
>
> Thanks for you help,
> Tibor Arpas

[Walter[_2_]]

Hi Tibor,
did you already contact Christian at www.streckenflug.at.
He showed us the glider routes in Austria.
Since the pilots from Vienna often fly in your country, they might be verry
willing to help to fight airspace restrictions
Regards and good success
Walter

"Tibor" > schrieb im Newsbeitrag
news:28104876.3949.1330968367200.JavaMail.geo-discussion-forums@vbbed8...
> Hi,
>
> are there any open/free tools to datamine on database of many IGC files?
>
> There is a threat that the free airspace for gliding in Slovakia is going
> to be significantly restricted. We need tools which help prepare arguments
> about airspace usage and frequency of our flights.
>
> If I was starting from scratch I would probably try to import all the
> flights into a GIS database (PostGIS) and do the data cleaning and queries
> there. Does anybody have experience with this?
>
> Thanks for you help,
> Tibor Arpas

[Paul Jessop[_2_]]

At 17:26 05 March 2012, Tibor wrote:
>Hi,
>
>are there any open/free tools to datamine on database of many IGC files?
>
>There is a threat that the free airspace for gliding in Slovakia is going
>to be significantly restricted. We need tools which help prepare
arguments
>about airspace usage and frequency of our flights.
>
>If I was starting from scratch I would probably try to import all the
>flights into a GIS database (PostGIS) and do the data cleaning and
queries
>there. Does anybody have experience with this?
>
>Thanks for you help,
>Tibor Arpas
>

Hi Tibor,

I have done some of this with the IGC files (about 8000 of them I recall)
from the British Gliding Association national ladder. It was published
recently in Sailplane and Gliding and I'll willingly help you in any way I
can. There are some animations to look at if you visit:

http://www.gliding.co.uk/sailplaneandgliding/recentarticles.htm#SG1

The same approach was used to look at all flights over FL100 to back up a
(successful) request for exemption from transponder carriage. I'll put this
on Flickr in a minute.

All this was done by brute force with PERL code, taking a day to run over
the IGC files. The visualisation was done in 3DEM which I highly recommend
for this task.

None of this is open source - though I'd be happy to open source my code if
it is any use.

Please contact me off list and I'll explain further.

Paul

[Max Kellermann]

Tibor > wrote:
> There is a threat that the free airspace for gliding in Slovakia is
> going to be significantly restricted. We need tools which help
> prepare arguments about airspace usage and frequency of our flights.

One could use the XCSoar calculation engine to extract data from bulk
IGC files. What exactly do you need?

The biggest problem with bulk IGC files is how to get so many IGC
files. The OLC is a commercial company and bans you for attempting to
download more than 10 files in one day. Therefore, supporting the OLC
is harmful for projects like yours.

Max

[Paul Jessop[_2_]]

At 17:26 05 March 2012, Tibor wrote:
>Hi,
>
>are there any open/free tools to datamine on database of many IGC files?
>
>There is a threat that the free airspace for gliding in Slovakia is going
>to be significantly restricted. We need tools which help prepare
arguments
>about airspace usage and frequency of our flights.
>
>If I was starting from scratch I would probably try to import all the
>flights into a GIS database (PostGIS) and do the data cleaning and
queries
>there. Does anybody have experience with this?
>
>Thanks for you help,
>Tibor Arpas
>

[Not sure why the subject didn't appear on this post before]

Hi Tibor,

I have done some of this with the IGC files (about 8000 of them I recall)
from the British Gliding Association national ladder. It was published
recently in Sailplane and Gliding and I'll willingly help you in any way I
can. There are some animations to look at if you visit:

http://www.gliding.co.uk/sailplaneandgliding/recentarticles.htm#SG1

The same approach was used to look at all flights over FL100 to back up a
(successful) request for exemption from transponder carriage. I'll put this
on Flickr in a minute.

All this was done by brute force with PERL code, taking a day to run over
the IGC files. The visualisation was done in 3DEM which I highly recommend
for this task.

None of this is open source - though I'd be happy to open source my code if
it is any use.

Please contact me off list and I'll explain further.

Paul

[Paul Jessop[_2_]]

At 17:26 05 March 2012, Tibor wrote:
>Hi,
>
>are there any open/free tools to datamine on database of many IGC files?
>
>There is a threat that the free airspace for gliding in Slovakia is going
>to be significantly restricted. We need tools which help prepare
arguments
>about airspace usage and frequency of our flights.
>
>If I was starting from scratch I would probably try to import all the
>flights into a GIS database (PostGIS) and do the data cleaning and
queries
>there. Does anybody have experience with this?
>
>Thanks for you help,
>Tibor Arpas
>

[Not sure why the subject didn't appear on this post before]

Hi Tibor,

I have done some of this with the IGC files (about 8000 of them I recall)
from the British Gliding Association national ladder. It was published
recently in Sailplane and Gliding and I'll willingly help you in any way I
can. There are some animations to look at if you visit:

http://www.gliding.co.uk/sailplaneandgliding/recentarticles.htm#SG1

The same approach was used to look at all flights over FL100 to back up a
(successful) request for exemption from transponder carriage. I'll put this
on Flickr in a minute.

All this was done by brute force with PERL code, taking a day to run over
the IGC files. The visualisation was done in 3DEM which I highly recommend
for this task.

None of this is open source - though I'd be happy to open source my code if
it is any use.

Please contact me off list and I'll explain further.

Paul

[Paul Jessop[_2_]]

At 17:26 05 March 2012, Tibor wrote:
>Hi,
>
>are there any open/free tools to datamine on database of many IGC files?
>
>There is a threat that the free airspace for gliding in Slovakia is going
>to be significantly restricted. We need tools which help prepare
arguments
>about airspace usage and frequency of our flights.
>
>If I was starting from scratch I would probably try to import all the
>flights into a GIS database (PostGIS) and do the data cleaning and
queries
>there. Does anybody have experience with this?
>
>Thanks for you help,
>Tibor Arpas
>

[Not sure why the subject didn't appear on this post before]

Hi Tibor,

I have done some of this with the IGC files (about 8000 of them I recall)
from the British Gliding Association national ladder. It was published
recently in Sailplane and Gliding and I'll willingly help you in any way I
can. There are some animations to look at if you visit:

http://www.gliding.co.uk/sailplaneandgliding/recentarticles.htm#SG1

The same approach was used to look at all flights over FL100 to back up a
(successful) request for exemption from transponder carriage. I'll put this
on Flickr in a minute.

All this was done by brute force with PERL code, taking a day to run over
the IGC files. The visualisation was done in 3DEM which I highly recommend
for this task.

None of this is open source - though I'd be happy to open source my code if
it is any use.

Please contact me off list and I'll explain further.

Paul

[Tibor]

Thanks everyone for suggestions, please keep them coming :). For now we need to answer simple question or two. e.g. how many "flights/hours have reached 7500ft MSL."?

We'll probably get only competition flights (no online competitions), as we luckily have quite enough competitions in Slovakia and it's easy to approach the organizers.

For longer term, of course we would like to have more sophisticated collection, filtering and ad hoc queries. That's why I mentioned importing flights into relational+spatial database. I think it's the propper groundwork for anybody doing statistics or visualizations. If we create something along this lines I'll let you know.

Thanks,
Tibor

On Monday, March 5, 2012 6:26:07 PM UTC+1, Tibor wrote:
> Hi,
>
> are there any open/free tools to datamine on database of many IGC files?
>
> There is a threat that the free airspace for gliding in Slovakia is going to be significantly restricted. We need tools which help prepare arguments about airspace usage and frequency of our flights.
>
> If I was starting from scratch I would probably try to import all the flights into a GIS database (PostGIS) and do the data cleaning and queries there. Does anybody have experience with this?
>
> Thanks for you help,
> Tibor Arpas



On Monday, March 5, 2012 6:26:07 PM UTC+1, Tibor wrote:
> Hi,
>
> are there any open/free tools to datamine on database of many IGC files?
>
> There is a threat that the free airspace for gliding in Slovakia is going to be significantly restricted. We need tools which help prepare arguments about airspace usage and frequency of our flights.
>
> If I was starting from scratch I would probably try to import all the flights into a GIS database (PostGIS) and do the data cleaning and queries there. Does anybody have experience with this?
>
> Thanks for you help,
> Tibor Arpas

[[email protected]]

Marrek has received 7121 IGC files from recent Slovakian competitions from Soaring Spot.

The easiest way to datamine for particular airspace is for sure with SeeYou Competition.

1. Create a task for each competition day (same task for everybody, something simple with two overlapping circles where a glider "starts" right off tow)

2. make an airspace file with only one airspace - the one that you are investigating

3. Run the Auto Evaluation on all 7000+ flights, save the CUC file (or make an XML if you wish)

4. search for "airspace violations", each one is listed in the CUC file. Description of the warning message is in the help file.

If you have a programmer who likes parsing this can all be very much automated ;)

Let me know if you need more help.
Andrej Kolar

On Tuesday, March 6, 2012 8:20:31 AM UTC+1, Tibor wrote:
> Thanks everyone for suggestions, please keep them coming :). For now we need to answer simple question or two. e.g. how many "flights/hours have reached 7500ft MSL."?
>
> We'll probably get only competition flights (no online competitions), as we luckily have quite enough competitions in Slovakia and it's easy to approach the organizers.
>
> For longer term, of course we would like to have more sophisticated collection, filtering and ad hoc queries. That's why I mentioned importing flights into relational+spatial database. I think it's the propper groundwork for anybody doing statistics or visualizations. If we create something along this lines I'll let you know.
>
> Thanks,
> Tibor
>
> On Monday, March 5, 2012 6:26:07 PM UTC+1, Tibor wrote:
> > Hi,
> >
> > are there any open/free tools to datamine on database of many IGC files?
> >
> > There is a threat that the free airspace for gliding in Slovakia is going to be significantly restricted. We need tools which help prepare arguments about airspace usage and frequency of our flights.
> >
> > If I was starting from scratch I would probably try to import all the flights into a GIS database (PostGIS) and do the data cleaning and queries there. Does anybody have experience with this?
> >
> > Thanks for you help,
> > Tibor Arpas
>
>
>
> On Monday, March 5, 2012 6:26:07 PM UTC+1, Tibor wrote:
> > Hi,
> >
> > are there any open/free tools to datamine on database of many IGC files?
> >
> > There is a threat that the free airspace for gliding in Slovakia is going to be significantly restricted. We need tools which help prepare arguments about airspace usage and frequency of our flights.
> >
> > If I was starting from scratch I would probably try to import all the flights into a GIS database (PostGIS) and do the data cleaning and queries there. Does anybody have experience with this?
> >
> > Thanks for you help,
> > Tibor Arpas

[Judy Ruprecht]

Hi, Tibor

You mention 7500' MSL as a key altitude... nationwide? or in certain
areas?

In either case, your text argument should mention the negative effect on
Slovakian participation in Internationally-recognized Federation
Aeronautique achievement awards. (eg: Gold & Diamond altitude & badge
completions... perhaps Silver as well, depending on takeoff/landing
elevation)

Ideally, this would be accompanied by some (impressive?) statistics on how
many soaring sites are affected, how many pilots have safely attained
Gold/Diamond altitude, both overall and in a recent period.

If the sticking point is Air Traffic Control "authorization" and/or
transponder equipment, you may find some help from the SSA (US) Government
Liaison Chair, Steve Northcraft, at

In the US, gliders as a "Class" are excluded from transponder requirements
that other aircraft must use at 10,000' MSL; there are also methods used
here to access Class A airspace, which for US gliders exists at and above
18,000' MSL.

Good Luck with your efforts!!

Judy

At 07:20 06 March 2012, Tibor wrote:
>Thanks everyone for suggestions, please keep them coming :). For now we
>nee=
>d to answer simple question or two. e.g. how many "flights/hours have
>reach=
>ed 7500ft MSL."?
>
>We'll probably get only competition flights (no online competitions), as
>we=
> luckily have quite enough competitions in Slovakia and it's easy to
>approa=
>ch the organizers.
>
>For longer term, of course we would like to have more sophisticated
>collect=
>ion, filtering and ad hoc queries. That's why I mentioned importing
>flights=
> into relational+spatial database. I think it's the propper groundwork
for
>=
>anybody doing statistics or visualizations. If we create something along
>th=
>is lines I'll let you know.
>
>Thanks,
>Tibor
>
>On Monday, March 5, 2012 6:26:07 PM UTC+1, Tibor wrote:
>> Hi,
>>=20
>> are there any open/free tools to datamine on database of many IGC
files?
>>=20
>> There is a threat that the free airspace for gliding in Slovakia is
>going=
> to be significantly restricted. We need tools which help prepare
>arguments=
> about airspace usage and frequency of our flights.
>>=20
>> If I was starting from scratch I would probably try to import all the
>fli=
>ghts into a GIS database (PostGIS) and do the data cleaning and queries
>the=
>re. Does anybody have experience with this?
>>=20
>> Thanks for you help,
>> Tibor Arpas
>
>
>
>On Monday, March 5, 2012 6:26:07 PM UTC+1, Tibor wrote:
>> Hi,
>>=20
>> are there any open/free tools to datamine on database of many IGC
files?
>>=20
>> There is a threat that the free airspace for gliding in Slovakia is
>going=
> to be significantly restricted. We need tools which help prepare
>arguments=
> about airspace usage and frequency of our flights.
>>=20
>> If I was starting from scratch I would probably try to import all the
>fli=
>ghts into a GIS database (PostGIS) and do the data cleaning and queries
>the=
>re. Does anybody have experience with this?
>>=20
>> Thanks for you help,
>> Tibor Arpas
>
>