M17 Project Forum

Full Version: Fragment LICHs
You're currently viewing a stripped down version of our content. View the full version with proper formatting.
Pages: 1 2
Hi All

I've just finished implementing M17 on the MMDVM, although it is almost completely untested in any meaningful way. I have included a network client for the reflector protocol, but it's hardcoded in the ini file currently, I will probably add a simple minded gateway with parrot at some point. I'll also add a gateway into APRS-IS once there is a standard for APRS/GPS data for M17.

One area that bothers me (apart from being able to distinguish between the link setup and normal data) is the handling of fragment LICH data.

You need to receive five frames to be able to recover the full LICH from the fragments. As an aside, I assume that an FN=0 is the first fragment, FN=1 is the second fragment, and then continuing in a round-robin fashion? This needs documenting.

My issue is with the integrity of the received fragment LICH data. In cases where it is needed for late entry, the overall checksum can be used to determine if you have a complete uncorrupted LICH, which is fine as far as it goes. The problem comes when receiving the fragments, some may have bit errors and you don't know which they are, so you try and rebuild the FICH with faulty data, and it fails. This could keep happening as you don't know which fragment is good or not. Would it not be better to dispense with the overall checksum on the LICH (saving 16-bits) and instead have a short checksum on each LICH fragment? That way we can be sure that each fragment is not corrupted, and can be used for the overall LICH without problems.

If that is not good enough, then maybe keep the overall checksum in the LICH and have the LICH spread over six frames, or more, instead of five to allow for the extra space for a local fragment checksum as above.

Jonathan  G4KLX
(10-19-2020, 11:17 AM)G4KLX Wrote: [ -> ]Hi All

I've just finished implementing M17 on the MMDVM, although it is almost completely untested in any meaningful way. I have included a network client for the reflector protocol, but it's hardcoded in the ini file currently, I will probably add a simple minded gateway with parrot at some point. I'll also add a gateway into APRS-IS once there is a standard for APRS/GPS data for M17.

One area that bothers me (apart from being able to distinguish between the link setup and normal data) is the handling of fragment LICH data.

You need to receive five frames to be able to recover the full LICH from the fragments. As an aside, I assume that an FN=0 is the first fragment, FN=1 is the second fragment, and then continuing in a round-robin fashion? This needs documenting.
This is what I did as a default from reading the spec.

My issue is with the integrity of the received fragment LICH data. In cases where it is needed for late entry, the overall checksum can be used to determine if you have a complete uncorrupted LICH, which is fine as far as it goes. The problem comes when receiving the fragments, some may have bit errors and you don't know which they are, so you try and rebuild the FICH with faulty data, and it fails. This could keep happening as you don't know which fragment is good or not. Would it not be better to dispense with the overall checksum on the LICH (saving 16-bits) and instead have a short checksum on each LICH fragment? That way we can be sure that each fragment is not corrupted, and can be used for the overall LICH without problems.

If that is not good enough, then maybe keep the overall checksum in the LICH and have the LICH spread over six frames, or more, instead of five to allow for the extra space for a local fragment checksum as above.

Jonathan  G4KLX

CRC isn't my thing, but each frame has its own CRC, right? So I thought the overall frame checksum not validating would indicate which LICH chunks were corrupt - so we'd get chunks 2,3,4,5,1; but 3 has a bad CRC check, so we have to wait until we get that chunk again to reconstruct.
Relayed for SP5WWP:

FN increases up to about 2^15, then yes, it rolls over. The most significant bit is used to tell if it's the last frame in a stream. Documentation to be made more clear on this soon.
(10-20-2020, 09:03 PM)KC1AWV Wrote: [ -> ]Relayed for SP5WWP:

FN increases up to about 2^15, then yes, it rolls over. The most significant bit is used to tell if it's the last frame in a stream. Documentation to be made more clear on this soon.

I'm not sure what that addresses? Both of those points are in the spec too, I think.
(aside: SP5WWP: fix your net!)
(10-20-2020, 08:57 PM)tarxvf Wrote: [ -> ]
(10-19-2020, 11:17 AM)G4KLX Wrote: [ -> ]Hi All

I've just finished implementing M17 on the MMDVM, although it is almost completely untested in any meaningful way. I have included a network client for the reflector protocol, but it's hardcoded in the ini file currently, I will probably add a simple minded gateway with parrot at some point. I'll also add a gateway into APRS-IS once there is a standard for APRS/GPS data for M17.

One area that bothers me (apart from being able to distinguish between the link setup and normal data) is the handling of fragment LICH data.

You need to receive five frames to be able to recover the full LICH from the fragments. As an aside, I assume that an FN=0 is the first fragment, FN=1 is the second fragment, and then continuing in a round-robin fashion? This needs documenting.
This is what I did as a default from reading the spec.

My issue is with the integrity of the received fragment LICH data. In cases where it is needed for late entry, the overall checksum can be used to determine if you have a complete uncorrupted LICH, which is fine as far as it goes. The problem comes when receiving the fragments, some may have bit errors and you don't know which they are, so you try and rebuild the FICH with faulty data, and it fails. This could keep happening as you don't know which fragment is good or not. Would it not be better to dispense with the overall checksum on the LICH (saving 16-bits) and instead have a short checksum on each LICH fragment? That way we can be sure that each fragment is not corrupted, and can be used for the overall LICH without problems.

If that is not good enough, then maybe keep the overall checksum in the LICH and have the LICH spread over six frames, or more, instead of five to allow for the extra space for a local fragment checksum as above.

Jonathan  G4KLX

CRC isn't my thing, but each frame has its own CRC, right? So I thought the overall frame checksum not validating would indicate which LICH chunks were corrupt - so we'd get chunks 2,3,4,5,1; but 3 has a bad CRC check, so we have to wait until we get that chunk again to reconstruct.

It's not clear about which fields are covered by the CRC. If it covers the encoded LICH fragment, that is great. That addresses my issue, and it means that I need to update my implementation also.

(10-21-2020, 09:43 AM)G4KLX Wrote: [ -> ]
(10-20-2020, 08:57 PM)tarxvf Wrote: [ -> ]
(10-19-2020, 11:17 AM)G4KLX Wrote: [ -> ]Hi All

I've just finished implementing M17 on the MMDVM, although it is almost completely untested in any meaningful way. I have included a network client for the reflector protocol, but it's hardcoded in the ini file currently, I will probably add a simple minded gateway with parrot at some point. I'll also add a gateway into APRS-IS once there is a standard for APRS/GPS data for M17.

One area that bothers me (apart from being able to distinguish between the link setup and normal data) is the handling of fragment LICH data.

You need to receive five frames to be able to recover the full LICH from the fragments. As an aside, I assume that an FN=0 is the first fragment, FN=1 is the second fragment, and then continuing in a round-robin fashion? This needs documenting.
This is what I did as a default from reading the spec.

My issue is with the integrity of the received fragment LICH data. In cases where it is needed for late entry, the overall checksum can be used to determine if you have a complete uncorrupted LICH, which is fine as far as it goes. The problem comes when receiving the fragments, some may have bit errors and you don't know which they are, so you try and rebuild the FICH with faulty data, and it fails. This could keep happening as you don't know which fragment is good or not. Would it not be better to dispense with the overall checksum on the LICH (saving 16-bits) and instead have a short checksum on each LICH fragment? That way we can be sure that each fragment is not corrupted, and can be used for the overall LICH without problems.

If that is not good enough, then maybe keep the overall checksum in the LICH and have the LICH spread over six frames, or more, instead of five to allow for the extra space for a local fragment checksum as above.

Jonathan  G4KLX

CRC isn't my thing, but each frame has its own CRC, right? So I thought the overall frame checksum not validating would indicate which LICH chunks were corrupt - so we'd get chunks 2,3,4,5,1; but 3 has a bad CRC check, so we have to wait until we get that chunk again to reconstruct.

It's not clear about which fields are covered by the CRC. If it covers the encoded LICH fragment, that is great. That addresses my issue, and it means that I need to update my implementation also.

Just had another look at specification, "Fig 6. An overview of the forward dataflow" to be precise, and it shows that the CRC only covers the FN and payload, not the fragment LICH. So my point still stands.
Quote:[quote pid="256" dateline="1603269830"]
It's not clear about which fields are covered by the CRC. If it covers the encoded LICH fragment, that is great. That addresses my issue, and it means that I need to update my implementation also.

Just had another look at specification, "Fig 6. An overview of the forward dataflow" to be precise, and it shows that the CRC only covers the FN and payload, not the fragment LICH. So my point still stands.


[/quote]

Whoops.  Blush 
Yep, agree. Something's gotta change.
The entire LICH is covered with a CRC. You need to decode all 5 LICH fragments and assemble them to check the CRC.

There are currently two ways to handle assembling the LICH.

1. grab 5 frames in sequence, check the CRC, if it does not match rotate the frames up to 5 time, computing the CRC each time.
2. keep the last 5 LICH frames received and compute the CRC. If you don't have a valid CRC, discard oldest frame and append the next one. Repeat until you get a valid CRC.

Or you could do some combination of 1 & 2.

The LICH has only incidental correlation with FN. The only requirement is that the LICH frames are sent in sequence and the FN increments sequentially from 0. LICH frames are not numbered. They only have logical ordering.
A somewhat parallel discussion occurring on github

Two potential issues have been identified.
  1. CRC check can't be checked until 5 LICH fragments have been received.
  2. FN is under the payload CRC and really isn't useful in reassembling LICH
Item 1 may result in delay in joining, but not necessarily a major flaw. If instead of a full LICH CRC we has a smaller CRC on each chunk, then it does improve likelihood of decoding in adverse situations, but I don't think the reduction in delay will be substantial. The Golay coding allows for correcting 3 bit errors and detecting up to 7 bit errors (wikipedia), thus already providing quite good protection and ability to reject a chunk. 

As pointed out above by G4KLX and WX9O and above and mobilinkd on github (maybe the same people), brute force seems the only way to reassemble at the moment since FN can't really be used without decoding the full payload. One option I thought of was changing the Golay to [23,12,7] and freeing 4 bits for each chunk that can be used to indicate the start of LICH. We'd want this to be coded but with 4 bits, the only real option is repetition coding.

Any downsides to the above? What am I missing?
Can I ask why there isn't any relationship between the FN and the LICH fragments? It seems logical to me to have the LICH fragment number connected to the FN to make for simpler processing. It doesn't cost anything in terms of adding extra bits to the protocol. Even if the FN is corrupt, it is still possible to add one to the last FN value and hope that the fragment is not too corrupt. Without such a relationship between the FN and LICH fragment, you are adding extra processing to the M17 implementation which is entirely avoidable. Therefore I would strongly suggest having a fixed relationship between FN value and the LICH fragment number.

If going to a Golay(23,12,7), I would suggest using the free'd bits for a local checksum rather than a LICH fragment number.

Jonathan G4KLX
(10-25-2020, 02:24 PM)G4KLX Wrote: [ -> ]Can I ask why there isn't any relationship between the FN and the LICH fragments?
Jonathan  G4KLX

Because the LICH is intended to be decoded without without the need to decode the payload.  And the FN exists in the payload.

Assuming a relationship between LICH and FN will likely result in problems joining a transmission after the FN rolls over.

I opened a separate issue on GitHub about the current need to decode the LICH to compute the payload CRC.  That has (or will) result in a revision to the spec.  The converse of the above is also true: it is not necessary to decode the LICH if one has a valid LSF; one need only decode the payload.
Pages: 1 2