... the user friendly GPS tool


Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Some NMEA file parsing problems
#1
Hello (1st time here Smile !
I've been playing with the data download from my Qstarz GPS data logger and with different programs I've been getting all kind of possible versions and precisions of either NMEA or GPX data. It seemed I could resolve all this format issues with RouteConverter, but then I discovered a bug when parsing NMEA file with RouteConverter. First entry of NMEA file wasn't parsed properly because of rounding errors when comparing values. It also seems that the comparison of entry's time to cover all entries which belong together was missing.

So I cloned the git repository, modified the code to my wishes and now I can read the NMEA file, convert it to GPX and then back to NMEA. The new NMEA file will have the same entries regarding the position/altitude/speed as the original NMEA file (no rounding errors). This of course requires some violation of the length of the fraction digits, but I think most modern GPS programs (not the hardware, I guess) are adaptable to different fraction digits in values. The length of the fraction digits can be specified in preferences/registry and if not, they get the default/original values.

The modified code is available on https://github.com/mrihtar/RouteConverter. For the moment it requires the directives for Maven to ignore test errors/failures (I know this is bad and I will fix the tests as soon as I am sure everything works OK):
Code:
-Dmaven.test.failure.ignore=true -Dmaven.test.error.ignore=true
Reply
#2
(31.07.2013, 15:19)mrihtar Wrote: Hello (1st time here Smile !

Hello mrihtar,

you're here for the first time and you're contributing code! Great!

(31.07.2013, 15:19)mrihtar Wrote: [..] but then I discovered a bug when parsing NMEA file with RouteConverter. First entry of NMEA file wasn't parsed properly because of rounding errors when comparing values. It also seems that the comparison of entry's time to cover all entries which belong together was missing.

Please provide me with a test file that shows the problem.

(31.07.2013, 15:19)mrihtar Wrote: The modified code is available on https://github.com/mrihtar/RouteConverter.

I've had a look at it and had a lot of comments since there are a lot of changes. I'd be happy to integrate your contribution but then I have to understand the implications of your changes. Generally a lot of small changes with well-understood implications are much easier to integrate than one big change.
--
Christian
Reply
#3
> Please provide me with a test file that shows the problem.

Code:
$GPZDA,100436,29,07,2013,,*44
$GPRMC,100436,A,4300.898329,N,00948.227878,E,0.0000,,290713,,A*4F
$GPGGA,100436,4300.898329,N,00948.227878,E,1,,,203.0821,M,,M,,*4B
$GPWPL,4300.898329,N,00948.227878,E,Position 1*60
$GPVTG,,T,,M,0.0000,N,0.0000,K,A*23
$GPZDA,100436,29,07,2013,,*44
$GPRMC,100436,A,4300.898328,N,00948.227878,E,0.0216,,290713,,A*4B
$GPGGA,100436,4300.898328,N,00948.227878,E,1,,,203.0060,M,,M,,*47
$GPWPL,4300.898328,N,00948.227878,E,Position 2*62
$GPVTG,,T,,M,0.0216,N,0.0400,K,A*22
$GPZDA,100437,29,07,2013,,*45
$GPRMC,100437,A,4300.898328,N,00948.227878,E,0.0000,,290713,,A*4F
$GPGGA,100437,4300.898328,N,00948.227878,E,1,,,203.0317,M,,M,,*45
$GPWPL,4300.898328,N,00948.227878,E,Position 3*63
$GPVTG,,T,,M,0.0000,N,0.0000,K,A*23
$GPZDA,100441,29,07,2013,,*44
$GPRMC,100441,A,4300.888168,N,00948.241986,E,10.4306,,290713,,A*79
$GPGGA,100441,4300.888168,N,00948.241986,E,1,,,203.0079,M,,M,,*48
$GPWPL,4300.888168,N,00948.241986,E,Position 4*63
$GPVTG,,T,,M,10.4306,N,19.3175,K,A*2B
$GPZDA,100446,29,07,2013,,*43
$GPRMC,100446,A,4300.887436,N,00948.263022,E,11.0955,,290713,,A*71
$GPGGA,100446,4300.887436,N,00948.263022,E,1,,,203.0009,M,,M,,*4E
$GPWPL,4300.887436,N,00948.263022,E,Position 5*64
$GPVTG,,T,,M,11.0955,N,20.5489,K,A*28

This file has 5 points with the following characteristics:
- first two points compared have the same time, but a slightly different positions
- second two points compared have the same position, but 1 second difference in time
- last two points are normal points (different position and time)

Official version of RouteConverter (v2.10) reads this file as 4 points only with some points/data (time, elevation) lost and duplicated time on last two points. Even if you feed the official version with a simpler NMEA file, it will behave the same (1. point problem, last points duplicated).

Of course there are much more combinations of entries in NMEA files which I didn't test, but at least with this fix I can process NMEA files that I got hands on.
Reply
#4
Hi Matjaz,

thank you very much. It helped me to find find some issues in the code. And fixing these issues does not some to require to write data with more digits. Please check for the upcoming commit that I'm currently preparing and testing.
--
Christian
Reply
#5
> thank you very much. It helped me to find find some issues in the code.

No problem - I am glad I could contribute something to make the SW better.

> And fixing these issues does not require to write data with more digits.

Well, this was probably not completely clear from my first post, as it's not the main cause for the NMEA parsing problem. My primary wish was to convert between formats without losing any digits of original data. The NMEA parsing problems just poped-up in the process, because I was doing the conversion with the NMEA format in the middle.

The primary problem of rounding errors can be easily demonstrated, if you open the GPX file, save it as NMEA; open the resulting NMEA and save it back to GPX and do the diff. With the increased internal precision this might go away, but I think you'll still need more digits in the GPX and NMEA file for the above process to retain all of the original digits without rounding errors. That's why I introduced additional registry entries for more digits.
Reply
#6
(02.08.2013, 17:15)mrihtar Wrote: My primary wish was to convert between formats without losing any digits of original data.

Actually that's a goal for RouteConverter, too. But I decided to stay within the limits of the file formats.

(02.08.2013, 17:15)mrihtar Wrote: The primary problem of rounding errors can be easily demonstrated, if you open the GPX file, save it as NMEA; open the resulting NMEA and save it back to GPX and do the diff.

With the increased internal precision this might go away

No, the problem is that NMEA and GPX positions need to be converted and that NMEA limits the number of digits.

(02.08.2013, 17:15)mrihtar Wrote: but I think you'll still need more digits in the GPX and NMEA file for the above process to retain all of the original digits without rounding errors. That's why I introduced additional registry entries for more digits.

Agreed, you need the additional digits. Please try to do a pull request against the latest commits.
--
Christian
Reply
#7
Today I've synced against the latest code in master branch. It seems that not all suggested changes made it yet to the code. This doesn't bother me much as long as I am able (= find time) to maintain my fork.

What I would really like to do is not to produce only the final jars, but the executable file itself (for Windows). Would you be so kind and help me build the .exe file? What tools/packages are you using to create it?
Reply
#8
(20.11.2013, 14:55)mrihtar Wrote: Today I've synced against the latest code in master branch. It seems that not all suggested changes made it yet to the code. This doesn't bother me much as long as I am able (= find time) to maintain my fork.

I thought I had integrated the patches that make sense for the RouteConverter release. What exactly is the diff between your branch and the master?

(20.11.2013, 14:55)mrihtar Wrote: What I would really like to do is not to produce only the final jars, but the executable file itself (for Windows). Would you be so kind and help me build the .exe file? What tools/packages are you using to create it?

You need to build under Windows and enable the launcher profile. That's all.
--
Christian
Reply
#9
I've double checked and it seems your pull requests slipped through. Sorry. I've reviewed the pull requests and found them too big and not what you intended to submit. I guess you've done the pull request from your master and updated it in the mean time? It's recommended to do each pull request from a separate branch (ideally with just a commit) to isolate the changes.
--
Christian
Reply


Forum Jump:


Users browsing this thread: 1 Guest(s)