Re: Logging CAN messages

Hi John,

Nice job. Fully agree with you that you have to do such a job in C and=20
forget Python.

Just one remark about the fake CAN message with the error flag. I guess=20
it would be good to use the so-called "all message mode" of the PIC32=20
controller. In that case you still have the information of the message=20
as it was received (maybe the correct identifier and for sure the=20
timestamp). Just add the error flag into the message on a bit which is=20
not used. This is what I implemented in my CANLOG command.

The PIC32 in my system (PIC32MX795) has 128 kB of RAM of which I can use=20
only 32 kB maximum for buffering, because of the existing firmware. By=20
using FIFO's of both CAN controllers and setting up the circular buffer=20
over these FIFO's it would be possible to store a maximum of 2048=20
messages. Maybe not enough, but it is a low cost solution.

Cheers

Kees

Op 3-11-2020 om 18:43 schreef John Dammeyer:
> Hi Kees,
> To make this project work I used 10Mbps SPI.  The Pi was the master.  Ori=
ginally on the Pi side I started with Python and then threw it into the tra=
sh bin pretty quickly.  Hard to debug and slow as molasses.
> Switching to C made all the difference.
>
> On the PIC32 side the system essentially recorded messages into a FIFO 40=
96 records long with each record 16 bytes in size.  The CAN interrupt routi=
ne just read out four 32 bit words into the record and incremented the queu=
e pointer.  The compiled C code was examined at the assembler level to ensu=
re that the code was as fast as hand coding it in assembler.  Same with rea=
ding it from the FIFO.
>
> At the other end, the SPI data is also blocked into 16 byte records and t=
he hardware set up so the master can do a block transfer of 16 bytes at ful=
l speed.   So if a 500kpbs CAN messages takes about 250 uS on the bus then =
the transfer of the 16 bytes to the Pi takes 1.6uS done purely by the hardw=
are.  An interrupt to the PIC32 when all 16 bytes were transferred.
>
> At the Pi end data written into files and a new file started after 4000 m=
essages were received.  Three IDs were created for GPS information.  Here's=
 a sample.
>
> STU Serial #000025  (Log #106)
> Date: 2018-02-16 16:28:02
> 01575000,16:28:02.893.75,0x18,0x7E,0x17,0x00,0x7F,0x61,0x00,0x00,0x03C
> 01575001,16:28:02.898.75,0x19,0x7E,0x17,0x00,0xA9,0x60,0x00,0x00,0x03C
> . . .
> 01575188,16:28:03.830.25,0xE1,0x69,0x25,0x1F,0x84,0xA3,0xFC,0xFD,0xFFD # =
GPS Pos:  52.2545633,-3.3774716
> 01575189,16:28:03.830.25,0xCA,0x07,0x00,0x00,0x00,0x00,0x00,0x00,0xFFE # =
GPS Alt: 199.4
> 01575190,16:28:03.830.25,0x12,0x02,0x10,0x10,0x1C,0x0D,0x00,0x00,0xFFF # =
GPS Time:2018/28/16-16:28:13.0
> . . .
>
> Post processing etc. all done after the fact since this was purely a logg=
ing application from my perspective.  Other programmers in the team dealt w=
ith sending the files up to the cloud and cleaning up folders.
>
> For my new client my only additional task will be to create 'fake' CAN ID=
s with ERROR flag information.  The client application will need to parse t=
he message files looking for these perhaps other messages and then can back=
 track as far as desired to determine what might have gone wrong.
>
> That's if they decide to go in this direction.
> John Dammeyer
>
>
>> -----Original Message-----
>> From: [email protected] [mailto:canlist-owner@vector-i=
nformatik.com] On Behalf Of Kees Zagers
>> Sent: November-03-20 2:25 AM
>> To: [email protected]
>> Subject: Re: [CANLIST] Logging CAN messages
>>
>> Hi John,
>>
>> Looks like a great product. The PIC32 is a nice controller for a CAN
>> front end system. Mainly the flexibility in the CAN controller makes it
>> very suitable.
>>
>> BTW I also added the multiple FIFO buffers (1024 messages) to the
>> firmware, to use it as front end for an RPi. The prototype worked,
>> unfortunately the customer did not get the order for production. For the
>> interface to the Pi we used the I2C, which is better standardized than
>> the SPI in my opinion. For the new CAN FD and upcoming XL however I2C
>> will not be fast enough.
>>
>> Kees
>>
>>
>> Op 1-11-2020 om 17:43 schreef John Dammeyer:
>>> Thanks Kees,
>>> I've passed on the information for your product to my client.  It looks=
 like I'm also going to be able to get one of these.  I've put a
>> photo on my web site since I don't remember if attachments work with thi=
s forum.
>>> http://www.autoartisans.com/CAN/STU-2.jpg
>>>
>>> It's got a PIC32 front end and connects via SPI to a PiZeroW and has GP=
S and RTC.  Starts logging c/w RTC based time stamps on
>> power up and once the Pi is alive and GPS TOD information is there it tr=
ansfers the files onto the Pi SD card.  Each file holds 4000
>> messages which include TOD, Lat/Long/Altitude once per second.
>>> The only change to the PIC32 firmware will be info 'faked' CAN messages=
 reporting bus error conditions mixed in with the regular
>> received CAN messages.  But that too will cost the customer so when I me=
et with them next week I'll know more.
>>> John Dammeyer
>>>
>>>
>>>
>>>> -----Original Message-----
>>>> From: [email protected] [mailto:canlist-owner@vector=
-informatik.com] On Behalf Of [email protected]
>>>> Sent: November-01-20 1:12 AM
>>>> To: [email protected]
>>>> Subject: RE: [CANLIST] Logging CAN messages
>>>>
>>>> Hello John,
>>>>
>>>> Just one remark. I was not suggesting that you (or your customer) woul=
d
>>>> make the change in the firmware. It would be an upgrade of the existin=
g
>>>> software and done by myself.
>>>>
>>>> Kees
>>>>
>>>>
>>>> John Dammeyer schreef op 31.10.2020 21:45:
>>>>> Hi Kees,
>>>>>
>>>>> That product is close to what they need.  Nicely done!
>>>>>
>>>>> Although when I had the scope on their system last Thursday it looks
>>>>> like the messages were running at least every 10mS.  I think they sai=
d
>>>>> 100Hz, 50Hz and 20Hz intervals.  I'd say I'd have to have the ability
>>>>> to store at least 5,000 messages at 500kbps.
>>>>>
>>>>> A few years ago I did do a similar logging project.  They wanted the
>>>>> data stored on a Raspberry PiZeroW and then upload to the cloud via
>>>>> WiFi when available.
>>>>>
>>>>> We found that no matter what we did the Pi still took about 12 second=
s
>>>>> to boot.  Ultimately what I did was design and build a PIC32 front en=
d
>>>>> to timestamp and queue all messages.  Once the Pi was awake the PIC32
>>>>> became an SPI slave to the Pi which then pulled up the stored
>>>>> messages.
>>>>>
>>>>> Unfortunately I don't have access to the hardware or legally the
>>>>> software.  I was hoping that there might be something that uses a
>>>>> KVaser or Port or some other USB dongle and a PC application that
>>>>> would do this.
>>>>>
>>>>> I'll talk to the client and mention your product.  Not that intereste=
d
>>>>> in making changes in Basic though.
>>>>>
>>>>> Thanks again
>>>>>
>>>>> John Dammeyer
>>>>>
>>>>> FROM: [email protected]
>>>>> [mailto:[email protected]] ON BEHALF OF Kees Zagers
>>>>> SENT: October-31-20 11:53 AM
>>>>> TO: [email protected]
>>>>> SUBJECT: Re: [CANLIST] Logging CAN messages
>>>>>
>>>>> Hi John,
>>>>>
>>>>> In principle I created this functionality in my SI2CBB
>>>>> (www.unicanner.com [1]). The embedded hardware has all the CAN
>>>>> functionality in this case. The CANLOG command I created has the even=
t
>>>>> logging possibility. Button (connected to one of the Arduino pins),
>>>>> specific message and error event can be defined by the user. The only
>>>>> restriction is the fact that I used the standard circular FIFO buffer
>>>>> of the built-in CAN controller of the Microchip PIC32. This buffer
>>>>> only contains 32 messages. So the pre-trigger information is 32
>>>>> messages, which probably will be a fraction of the messages during th=
e
>>>>> last five seconds. The post-trigger information can be as long as you
>>>>> want. The PIC32 has the possibility to create 32 of these FIFO buffer=
s
>>>>> in RAM, where every FIFO can be linked to one or more filters and
>>>>> masks. If all FIFO's are linked to the same filter and mask, they can
>>>>> be cascaded, which gives us a buffer of 1024 messages. The problem
>>>>> however in this case it is not circular; if the last FIFO is full it
>>>>> stops. I have to check, but it should be possible to detect the buffe=
r
>>>>> full interrupt of the last FIFO and return to the first FIFO again. I=
n
>>>>> that case we can have at least 1 second of the pre-trigger messages i=
n
>>>>> a normal CANbus system.
>>>>>
>>>>> Anyway it is difficult to say how large the circular buffer should be
>>>>> to contain the information of 5 or 10 seconds, because it depends of
>>>>> course on the bus speed and bus load.
>>>>>
>>>>> The result of the logging can be directly uploaded to a PC through th=
e
>>>>> USB interface or locally stored on an SD flash card.
>>>>>
>>>>> So far my input.
>>>>>
>>>>> Cheers
>>>>>
>>>>> Kees
>>>>>
>>>>> Op 30-10-2020 om 06:50 schreef John Dammeyer:
>>>>>
>>>>>> I haven't kept up to date with newer developments over the last few
>>>>>> years. My Lawicel CANUSB, RM CANUSB and monitor program plus
>>>>>> whatever I've written has been adequate for the tasks I do.
>>>>>>
>>>>>> I've been asked to determine if a tool exists that can log CAN
>>>>>> messages in a circular buffer. Say 10 seconds worth and when a
>>>>>> trigger like either a message or say an error flag occurs the
>>>>>> logging to the circular buffer goes for 5 seconds and then is
>>>>>> stopped and saved.
>>>>>>
>>>>>> Essentially the system shows random faults and they'd like to know
>>>>>> what happened for 5 seconds on either side of the event.
>>>>>>
>>>>>> The even can be: a contact closure like a button pressed by an
>>>>>> operator; a specific message, Bus error as signaled by an error
>>>>>> flag.
>>>>>>
>>>>>> Does a tool like that exist?
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> John Dammeyer
>>>>>>
>>>>>> "ELS! Nothing else works as well for your Lathe"
>>>>>>
>>>>>> Automation Artisans Inc.
>>>>>>
>>>>>> www dot autoartisans dot com
>>>>> --
>>>>>
>>>>> Kees Zagers
>>>>>
>>>>> SI-KWADRAAT B.V.
>>>>>
>>>>> Van Kemenadelaan 13
>>>>>
>>>>> 5674 PP  NUENEN
>>>>>
>>>>> Tel. +31(0)40-2839582
>>>>>
>>>>> [email protected]
>>>>>
>>>>>
>>>>> Links:
>>>>> ------
>>>>> [1] http://www.unicanner.com
>>>> --
>>>> Archives and useful links: http://groups.yahoo.com/group/CANbus
>>>> Subscribe and unsubscribe at www.vector.com/canlist/
>>>> Report any problems to <[email protected]>
>>> --
>>> Archives and useful links: http://groups.yahoo.com/group/CANbus
>>> Subscribe and unsubscribe at www.vector.com/canlist/
>>> Report any problems to <[email protected]>
>> --
>> Archives and useful links: http://groups.yahoo.com/group/CANbus
>> Subscribe and unsubscribe at www.vector.com/canlist/
>> Report any problems to <[email protected]>
> --
> Archives and useful links: http://groups.yahoo.com/group/CANbus
> Subscribe and unsubscribe at www.vector.com/canlist/
> Report any problems to <[email protected]>

--
Archives and useful links: http://groups.yahoo.com/group/CANbus
Subscribe and unsubscribe at www.vector.com/canlist/
Report any problems to <[email protected]>