Scanning RS485 Stream

techElder

Well-known member
Wow! This can drive you nuts! RS485 has stubs that "listen" to the message traffic on the pair. I'm building one of the "slaves/stubs" that sit there and wait for their own address with instruction to do something.

I didn't quite realize that EVERY message that passes on the pair has to be decoded to see if it is for this particular slave. EVERY message. If there is 250 stubs, EVERY one of them is doing the same thing ... scanning EVERY message.

Then to watch for hser interrupts and flags and work through the scratchpad memory to see if there is a message with the right address. WHEW!

Can't lose any characters in the stream, so have to keep up with the pointers.

I'm going to go crazy before I get this right.

Code:
[color=Blue]do
      if [/color][color=Purple]@ptr [/color][color=DarkCyan]= [/color][color=Black]QUALIFIER [/color][color=Blue]then
            [/color][color=Black]messageHead [/color][color=DarkCyan]= [/color][color=Purple]ptr
            [/color][color=Blue]inc [/color][color=Purple]ptr
            [/color][color=Black]messageAddress [/color][color=DarkCyan]= [/color][color=Purple]@ptr
            [/color][color=Blue]if [/color][color=Black]messageAddress [/color][color=DarkCyan]<> [/color][color=Black]ZERO [/color][color=DarkCyan]AND [/color][color=Black]messageAddress [/color][color=DarkCyan]<> [/color][color=Black]stubAddress [/color][color=Blue]then
                  [/color][color=Purple]ptr [/color][color=DarkCyan]= [/color][color=Black]lastByte [/color][color=DarkCyan]+ [/color][color=Navy]1
                  [/color][color=Blue]exit
            endif
            inc [/color][color=Purple]ptr
            [/color][color=Black]messageFunction [/color][color=DarkCyan]= [/color][color=Purple]@ptr
            [/color][color=Blue]inc [/color][color=Purple]ptr
            [/color][color=Blue]if [/color][color=Purple]@ptr [/color][color=DarkCyan]= [/color][color=Blue]LF then
                  [/color][color=Black]messageTail [/color][color=DarkCyan]= [/color][color=Purple]ptr
                  [/color][color=Blue]inc [/color][color=Purple]ptr
                  [/color][color=Blue]exit
            endif
      else
            inc [/color][color=Purple]ptr
      [/color][color=Blue]endif
loop until [/color][color=Purple]ptr [/color][color=DarkCyan]> [/color][color=Black]lastByte[/color]
 

inglewoodpete

Senior Member
If you have any control over the source of the data, then keep it at the slowest rate possible (baud and repetition rates). During development, have a dedicated pin indicate when the chip is busy processing incoming RS485. If you have an oscilloscope, you can then get a feel for how busy the 485 slave is.

Just think: most micros just sit in a loop waiting for an input. You're reducing that wastage. :)

I'm going to go crazy before I get this right.
The forum has voting buttons. Perhaps you should get forum members to vote on how crazy you are before you start. Imagine going to your analyst with only half the data he needs to lock you away in a padded cell? ;)
 

techElder

Well-known member
Crazy?

IWP, you must have been there; done that! :)

“A question that sometimes drives me hazy: am I or are the others crazy?” &#8213; Albert Einstein​
“The Edge ... There is no honest way to explain it because the only people who really know where it is are the ones who have gone over.” &#8213; Hunter S. Thompson​
“When you are crazy you learn to keep quiet.” &#8213; Philip K. Dick​
“Boredom can be a lethal thing on a small island.” &#8213; Christopher Moore​
“Being crazy isn't enough.” &#8213; Dr. Seuss​
 

techElder

Well-known member
The point I'm pondering now is whether to put this scanning within the Interrupt sub or in the main loop.

I know ... it depends on how fast and how much, but its still a point to consider.

Its been done both ways in examples here on the forum by everyone that's commented (and sometimes both ways by the same guru ;) !)

I don't think my project will have anywhere near the traffic that a commercial RS485 system would have, and messages will be no longer than 10 or 12 characters. I'm thinking that when I test I just won't see any difference, but will still have to be tried both ways.

Sorry for just wandering around ...
 

hippy

Technical Support
Staff member
The point I'm pondering now is whether to put this scanning within the Interrupt sub or in the main loop.
From experience, interrupts won't help you and will probably make things much harder. Scanning an input stream requires a dynamic state machine and it is easier to keep all notions of state within the code handling the stream than try and separate it out for every byte received.

Also, you are not guaranteed to get an interrupt for every byte received if still processing data within that interrupt while more byes arrive. So you need to implement some non-interrupt means to check for bytes, and it soon becomes clear that in doing so the interrupts are unnecessary and are just over-complication.

What you are wanting to do should not be too hard but the main thing is having an elegant solution to handling it. The following ( untested, but similar to what I have done myself ) would check for a packet in the form "STX <ID> <cmd> ETX", The 'ptrNext' variable is used to note where the next packet check should be from, which will be at the start of the stream less the first byte, if a valid packet is not found, after the packet just processed if it is valid. That ensures nothing will be missed even if the data is corrupted or the receiver desynchronises -

Code:
Do
  Gosub HandlePacket
Loop

HandlePacket:
  ptrNext = ptr + 1
  Gosub ThisPacket
  ptr = ptrNext
  Return

ThisPacket:
  ; Check packet validity first
  Gosub GetByteInB0 : If b0 <> STX Then : Return : End If
  Gosub GetByteInB0 : id  = b0
  Gosub GetByteInB0 : cmd = b0
  Gosub GetByteInB0 : If b0 <> ETX Then : Return : End If
  ; Handle packet only after checking validity
  If id = MY_ID Then
    Gosub ObeyCmd
  End If
  ptrNext = ptr
  Return

GetByteInB0:
  Do : Loop While ptr = hSerPtr
  b0 = @ptrInc
  Return
You can inline the 'GetByteInB0' routine, even inline everything and use appropriate GOTO's to make things more efficient, but like this would be a good starting point. Get it working then optimise, or it makes it harder to get functionality correct and harder to test and debug.
 

hippy

Technical Support
Staff member
This is probably the optimised in-line code for the above example -

Code:
Do
  Do

    Do
      Do : Loop While ptr = hSerPtr
    Loop Until @ptrInc = STX

    ptrNext = ptr

    Do : Loop While ptr = hSerPtr
    id = @ptrInc

    Do : Loop While ptr = hSerPtr
    cmd = @ptrInc
  
    Do : Loop While ptr = hSerPtr
    If @bPtrInc <> ETX Then Exit
        
    If id = MY_ID Then
      Gosub ObeyCmd
    End If

  Loop

  ptr = ptrNext

Loop
That allows checksumming to be added quite easily if the packet contains a checksum, something like ...

Code:
   Do : Loop While ptr = hSerPtr
   chk = @ptrInc
   b0 = id ^ cmd ^ $FF
   If chk <> b0 Then Exit
 
Last edited:

Buzby

Senior Member
Hi hippy,

I'm not totally in agreement with the logic in your code.

The lines like "Do : Loop While ptr = hSerPtr" will stop the code doing anything else until a char is RXed. This means if the Master stops TXing, or a wire breaks, then the Slave is stuck in a tight loop waiting for a char which will never come, and can't do anything else.

Two options to avoid this are (1) put this code in a second task, (2) re-write the code to eliminate tight loops which are dependant on external events.

My preference would be (2), as I only have a one-track mind !.

Cheers,

Buzby
 

techElder

Well-known member
Glad to hear agreement on not having the hser interrupt working this. I was having trouble getting it coordinated with having two hardware interrupts that are not in the same time frame, but the logic of it all was going over my head.

I usually put some kind of counter in a tight loop, but I also don't want to design something that is guaranteed to "sometimes" miss characters. :)

My comms will have 1 second timeouts as a last resort, but a "stuck" loop means I'd have to have a more complex reset function.
 

hippy

Technical Support
Staff member
If the slave has to do something else while not receiving data then you can possibly call that code from the 'GetByteInB0' routine, within the waiting for data loop.

Interrupting the stream handling to do something else may be a more appropriate way to handle what else it has to do, especially if that is time based.

A full description of what you need, what the application has to do, will make it easier to suggest what the best solution may be.
 

techElder

Well-known member
Yeah, I understand what you're saying about more specific suggestions, but I'm happy with what you folks have said so far. It has pushed me out of a hole I dug. ;)

There's almost nothing being done while waiting for a command to do something.

The reason for the hserial interrupt was to give a "GO!" signal. The RS485 line is quiet beforehand by design, and then something/anything is sent. That generates the hserial interrupt which causes something to happen within a very short time.
 

hippy

Technical Support
Staff member
You could actually craft a state machine which might not be a complicated as I'd first suggested but would depend on exactly what packet data you have. My experience is mostly with MIDI which would have a huge number of states because the data packets are all variable.

This would be quite simple for my "STX <id> <cmd> ETX" packet example. Call "HandleStream" whenever you want to ...

Code:
state = STATE_STX
Do
  Gosub DoWhatever
  Gosub HandleStream
Loop

HandleStream:

  If ptr <> hSerPtr Then
    Select Case state

      Case STATE_STX:
        If @ptrInc = STX Then
          ptrNext = ptr
          state = STATE_ID
        End If

      Case STATE_ID:
        id = @ptrInc
        state = STATE_CMD

      Case STATE_CMD:
        cmd = @ptrInc
        state = STATE_ETX

      Case STATE_ETX:
        If @ptrInc <> ETX Then
          ptr = ptrNext
        Else
          If id = MY_ID Then
            Gosub ObeyCmd
          End If
        End If
        state = STATE_STX

    End Select
  End If
  Return
That can also be optimised to something more efficient.

I would guess you could code that so that it suspends when there is no more data and is only reactivated when there is more data but I don't know if it would really be of any benefit. Again it depends on exactly what your program has to do.
 

techElder

Well-known member
Hippy, after comparing your code in #6 above with what I was doing, I couldn't explain why you were resetting "ptrNext = ptr" and then "ptr = ptrNext". Seems like a lot of rescanning of the scratchpad.

Perhaps a mistype?

I was expecting ptr to be set after the scanned part of scratchpad.

EDIT: Never mind, Hippy. You already told me in #5 above, but the positioning is still not totally clear.
 
Last edited:

BeanieBots

Moderator
I had a similar issue about six months ago with a bank of PICAXEs all listening and replying to a VB app.
The PICAXEs primary functions were servo loops so timing was important whilst also listening for new commands.
I started by using the hser interrupt but eventually gave up mainly because too much time was being spent in the interrupt routine.
I then adopted a simple catch within the servo loops.
"do pause 30: while ptr <> hserptr"
The pause was to allow enough of the command to come in to make it worth bothering to start building the new 'state'
Fortunately, my commands all started and ended with distinct characters so it was easy to pop everything into ram using bptr and then walk backwards through bRAM after the message end was received. There was also a trap to clear the state if too many characters were received.
After a complete message was received, everything related was handled within the do:loop.
This approach allowed the slaves to carry on their main objective with no hangs even when bombarded with duff data and/or no data.
 

techElder

Well-known member
So, BB, you ended up copying serial into RAM until you received a ETX of some sort? Then you used a scan of RAM to adjust your equipment? You didn't provide enough sample for me to see.

My messages also have clear STX and ETX characters and a defined field. There is the possibility of more characters as data, but only a few bytes.

Were you concerned with getting trapped into a tight loop waiting for characters that don't get sent?
 

techElder

Well-known member
The hole I dug myself into was the specification of the "packet" or "message".

I've specified a variable length data field which just makes it too complicated when the end of the message doesn't get there.

Its easy to find the end of the message when its there.

So, I've revised the spec to include a fixed length, four digit data field.

Now I know when the end isn't here. ;)
 

hippy

Technical Support
Staff member
EDIT: Never mind, Hippy. You already told me in #5 above, but the positioning is still not totally clear.
Once synchronised, the next start will be what should be the STX of the next packet. The re-scanning is only done when the packet is not valid.

In that case we keep throwing away the first byte of the last stream and trying again. That gets around issues where the data in the packet may have an STX value.

That did make me realise there could still be a potential mis-synchronisation issue. Consider a packet of "STX id cmd dat ETX". If cmd=$03(ETX) and dat=$02(STX) the packet could mis-synchronise if it missed the first genuine STX sent -

Code:
Intended         id  cmd dat         id  cmd dat
                  |   |   |           |   |   |
Transmited   STX $00 $03 $02 ETX STX $00 $03 $02 ETX
                          |   |   |   |   |
Matched                  stx id  cmd dat etx
                              |   |   |
Actioned                     $03 $02 $00
I am not sure what the solution to that is but adding extra packet framing bytes and checksumming may mitigate the issue.
 

techElder

Well-known member
In my case I'm sending only limited ASCII values (mostly because the packet needs to be visible on a terminal.)

I've been thinking about the "stuck in a tight loop with no characters being sent" problem and how to solve.

This macro of your GETBYTEINTOB0 sub came to mind:

Code:
[color=Navy]#macro [/color][color=Black]GetSerByteTimeout[/color][color=Blue]([/color][color=Purple]b0[/color][color=Black],errFlag[/color][color=Blue])
      [/color][color=Green]' Enter with errFlag = 0, TIMEOUT = come error code
      ' Use settimer t1s_"MHZ" before entering for 1 second timeout; toFlag will be 0.
      [/color][color=Blue]Do
            if [/color][color=Purple]toFlag [/color][color=DarkCyan]= [/color][color=Navy]1 [/color][color=Blue]then
                  [/color][color=Black]errFlag [/color][color=DarkCyan]= [/color][color=Black]TIMEOUT [/color][color=Green]; defined constant
                  [/color][color=Blue]exit
            endif
      Loop While [/color][color=Purple]ptr [/color][color=DarkCyan]= [/color][color=Purple]hSerPtr
      b0 [/color][color=DarkCyan]= [/color][color=Purple]@ptrInc[/color]
[color=Navy]#endmacro[/color]
 

hippy

Technical Support
Staff member
I cannot really see what advantage a timeout would have. And simply exiting from the routine when there isn't any data, having error flags and the like, is, in my mind, only likely to complicate things. I believe a good rule of thumb is that if it is not elegant then it is probably not the right way to do it.

I think you may be getting hung up on 'how to do it' rather than looking at the big picture and considering 'what needs to be done' and then choosing a solution which is the best way to achieve that. Without knowing the whole though it is hard to advise on one specific part of it. It is not clear what perceived problem you are trying to overcome.
 

techElder

Well-known member
Don't blame me! ;) ;) Buzby made me think like that in #7 above! :) I'm skeered of bein' stuck! :)

I know what you mean, though. Some problems are better solved when you find them.

I'm coding and adjusting specs as I go.

Thanks, Hippy!
 

hippy

Technical Support
Staff member
I'm coding and adjusting specs as I go.
It's always better to design first, then code second. Though that is never practised as much as it is preached, it can help avoid getting into a maze, pushing forward and getting into more difficulty.

It occurs to me that the state machine implementation could be worked into a byte received interrupt ...

Code:
Interrupt:
  Do
    hSerFlag = 0
    Gosub HandleStream
  Loop Until hSerFlag = 0
  SetIntFlags %00100000
  Return
 

rossko57

Senior Member
To avoid reinventing wheels, and also offering the possibility of co-opting in existing code; there exist many robust commercial protocols for this kind of traffic. This task looks like a good fit for Pelco-D as used in CCTV wired controls. There's at least one Pelco on Picaxe project around here somewhere.

Modbus is probably over the top, but reading up on that serial protocol might offer some insights on timeouts - the protocol specifies maximum inter-character gaps and minimum inter-message gaps, all part of making it solid.

You might also consider a watchdog style implementation for catching "no comms". A timer is setup to interrupt in N milliseconds. When successful comms is detected, the timer is reset and should never activate. It depends what sort of action you want to take in the case of failure, whether that's any more use or easier to implement than a more conventional timeout, or even something to be done alongside.
 

techElder

Well-known member
The protocol I'm using is a modified Modbus ASCII protocol. I liked that PelcoD used 4-bytes that rendered bits for hardware functions. I'm allowing for that, but also for the bytes to be data values as sent.

I'm using an LRC checksum from Modbus when I send data, but not for every packet, just yet. Time will tell, because I have a very limited transmission range. I don't think I'll need the same things needed on a 1000 meter loop.

There you go, rossko57, agreeing with Buzby about "tight loops" and timeouts. :)
 

rossko57

Senior Member
It ain't really Modbus-ish unless there is handshaking normally going on - Modbus master expects a response. Slave won't respond to corrupted packets. Lack of response (another timeout in action!) allows for the master to attempt recovery/retry.

I didn't fully understand earlier comments about "GO" signalling; but in looking at Modbus you may also have noticed the "broadcast" mode. Master sends packet with no address specified; ALL slaves act on this (but do not respond). Might be usefully adapted if this "GO" requires synchronising across your slaves.
 

BeanieBots

Moderator
In reply to post#15, yes, my exact issue was to not have any PICAXE hanging around waiting for data that might never come.
All my serial data is via Bluetooth, so the possibility of no and/or corrupt data is very real.
Also, each of the slave PICAXEs needed to run servo loops (admittedly at only 1Hz) so they could not spend much time doing serial data decoding.

The rough sequence is as follows:-

The main servo loop code is within an infinite do:loop
Also within that loop, is a check "if ptr <> hserptr" to check is anything has been received.
If something has been received, wait a bit for at least a few more characters to arrive.
All the time [ptr <> hserptr], put the characters into RAM using bptr but STOP if more than the maximum number ever expected has been reached.
In your case, it is even easier because you have a clear start,end and now you even have fixed length.
My problem was that I did not (always) have time to process the received data which is why I put it into RAM. That way I could still receive data into scratchpad for processing next time around the loop whilst still holding data pointed to by bptr to be acted apon when time permitted.

It's a little messy and probably over convoluted but it's been working reliably for several months now with data being sent via Ethernet to a vb app then via Bluetooth to multiple PICAXEs and responses coming back via the same path. Meanwhile, none of the PICAXEs have ever stopped doing what they should be doing even when the VB app is down or sending a complete set of new parameters to all PICAXEs.
It sounded just like your issue which is why I mentioned my approach.
Sorry I can't explain it a bit better.

The state engine part is really simple.
0= start is received.
1=data coming in.
2=end received
3=too many characters received
4=waiting for next start character. (gets set after data has been processed)

It does not catch if an entire packet has been missed.
This is handled on the VB side by having the PICAXE send an ACK to each packet. No ACK and the packet is simply sent again.
 

techElder

Well-known member
Thanks very much, BB. That is very clear to me. The counter within an "if ptr=hserptr" loop is what I've implemented before, and seems to be a possibility to add now.

But to hippy's thinking, until I see the problem in my project during testing, I'm going to hold back on getting "over convoluted."

Your solutions interest me greatly, because I'm planning to add RF in some way; perhaps Bluetooth.

Thanks!
 

BeanieBots

Moderator
Glad I could help.
Whichever method you use will really depend on what else the PICAXE needs to do.
Without thinking, I wrote pages of code and ended up getting more errors than good packets.
This was because like a fool I had the poor old PICAXE doing PWM,One-Wire and I2C comms and still expected background serial to work reliably.
I eventually got it all working 100% reliably by assigning the right group of tasks to the right slave PICAXEs running at appropriate clock speeds.
(I needed to include handshaking for the I2C as mentioned in another thread a while back)
My serial comms does not use any checksums as I don't have enough overhead to do it but I did throw in a couple of fixed characters at fixed character counts for added security/comfort.
If you are going to add an RF element, I strongly suggest an "intelligent" type that does a lot of data integrity and error checking for you. The cheap HC-05/HC-06 work really well for Bluetooth.
 

techElder

Well-known member
WORD pointer for scratchpad?

I finally got back to optimizing my serial functions and quickly ran into a problem on my 40X2.

The variable ptr is a byte pointer while the pointer hserptr can go to 1023.

Therefore, the comparison of the two quickly find a failure point in my ring buffer.

Am I missing a WORD pointer (ptr) for the scratchpad?


This is probably the optimised in-line code for the above example -

Code:
Do
  Do

    Do
      Do : Loop While ptr = hSerPtr
    Loop Until @ptrInc = STX

    ptrNext = ptr

    Do : Loop While ptr = hSerPtr
    id = @ptrInc

    Do : Loop While ptr = hSerPtr
    cmd = @ptrInc
  
    Do : Loop While ptr = hSerPtr
    If @bPtrInc <> ETX Then Exit
        
    If id = MY_ID Then
      Gosub ObeyCmd
    End If

  Loop

  ptr = ptrNext

Loop
That allows checksumming to be added quite easily if the packet contains a checksum, something like ...

Code:
   Do : Loop While ptr = hSerPtr
   chk = @ptrInc
   b0 = id ^ cmd ^ $FF
   If chk <> b0 Then Exit
 

hippy

Technical Support
Staff member
The 'ptr' variable is intended to be the same size as 'hserptr' and should be on the 40X2.

Code:
#Picaxe 40x2
#Terminal 9600
Do
  Sertxd( "---", Cr, Lf )
  ptr = 0       : Sertxd( "Expected 0    Got ", #ptr, Cr, Lf )
  ptr = ptr - 1 : Sertxd( "Expected 1023 Got ", #ptr, Cr, Lf )
  ptr = ptr + 1 : Sertxd( "Expected 0    Got ", #ptr, Cr, Lf )
  b0  = @ptrdec : Sertxd( "Expected 1023 Got ", #ptr, Cr, Lf )
  b0  = @ptrinc : Sertxd( "Expected 0    Got ", #ptr, Cr, Lf )
  ptr = $FFFF   : Sertxd( "Expected 1023 Got ", #ptr, Cr, Lf )
  Pause 1000
Loop
That works as expected under simulation so might be worth testing on your hardware.

And check that 'ptrNext' isn't defined as a byte variable.
 

techElder

Well-known member
I was just watching a SERTXD display of ptr next to hserptr.

ptr rolled over at 255 and hserptr kept on going.

Code:
Loop1  ptr: 248  hSerPtr: 813
Loop1  ptr: 249  hSerPtr: 813
Loop1  ptr: 250  hSerPtr: 814
Loop1  ptr: 251  hSerPtr: 815
Loop1  ptr: 252  hSerPtr: 816
Loop1  ptr: 253  hSerPtr: 816
Loop1  ptr: 254  hSerPtr: 817
Loop1  ptr: 255  hSerPtr: 818
Loop1  ptr: 0  hSerPtr: 819
Loop1  ptr: 1  hSerPtr: 819
Loop1  ptr: 2  hSerPtr: 820
Loop1  ptr: 3  hSerPtr: 821
Loop1  ptr: 4  hSerPtr: 821
Loop1  ptr: 5  hSerPtr: 822
Loop1  ptr: 6  hSerPtr: 823
Loop1  ptr: 7  hSerPtr: 823
 
Last edited:

techElder

Well-known member
Hmmmmm, your code snippet works fine on my hardware.

How in the heck can I be re-defining ptr to be a byte variable?


EDIT: Never mind! :) You already fixed my problem in post #31. My 'ptrNext' is a byte variable and limiting. Thanks Hippy! Again!
 

techElder

Well-known member
Works a treat with the ptr chasing the hserptr around the ring buffer. I do have to zero out the qualifier in my application. In overscanning the buffer, I was finding the "leftover" packets before the new packets, but that works just fine.

No serial interrupts. I sure enjoyed deleting that code! :)
 

Buzby

Senior Member
I wish there was a way to set the size of the hserptr, to nibble, byte or word, as needed.

A 1K serial buffer is HUGE, and many times I have to jump hoops to use just part of the scratchpad as serial buffer, and the rest for other uses, especially I2C.
 

BeanieBots

Moderator
I wish there was a way to set the size of the hserptr, to nibble, byte or word, as needed.

A 1K serial buffer is HUGE, and many times I have to jump hoops to use just part of the scratchpad as serial buffer, and the rest for other uses, especially I2C.
Seconded. I'm having a great struggle with that very issue right now.
It would also be very nice to have the ability to have more than one pointer when using scratchpad for more than one use.
 
Top