RE: Logging CAN messages

Hi Kees,
To make this project work I used 10Mbps SPI.  The Pi was the master.  Origi=
nally on the Pi side I started with Python and then threw it into the trash=
 bin pretty quickly.  Hard to debug and slow as molasses.=20
Switching to C made all the difference. =20

On the PIC32 side the system essentially recorded messages into a FIFO 4096=
 records long with each record 16 bytes in size.  The CAN interrupt routine=
 just read out four 32 bit words into the record and incremented the queue =
pointer.  The compiled C code was examined at the assembler level to ensure=
 that the code was as fast as hand coding it in assembler.  Same with readi=
ng it from the FIFO.

At the other end, the SPI data is also blocked into 16 byte records and the=
 hardware set up so the master can do a block transfer of 16 bytes at full =
speed.   So if a 500kpbs CAN messages takes about 250 uS on the bus then th=
e transfer of the 16 bytes to the Pi takes 1.6uS done purely by the hardwar=
e.  An interrupt to the PIC32 when all 16 bytes were transferred.

At the Pi end data written into files and a new file started after 4000 mes=
sages were received.  Three IDs were created for GPS information.  Here's a=
 sample.

STU Serial #000025  (Log #106)
Date: 2018-02-16 16:28:02
01575000,16:28:02.893.75,0x18,0x7E,0x17,0x00,0x7F,0x61,0x00,0x00,0x03C
01575001,16:28:02.898.75,0x19,0x7E,0x17,0x00,0xA9,0x60,0x00,0x00,0x03C
. . .
01575188,16:28:03.830.25,0xE1,0x69,0x25,0x1F,0x84,0xA3,0xFC,0xFD,0xFFD # GP=
S Pos:  52.2545633,-3.3774716
01575189,16:28:03.830.25,0xCA,0x07,0x00,0x00,0x00,0x00,0x00,0x00,0xFFE # GP=
S Alt: 199.4
01575190,16:28:03.830.25,0x12,0x02,0x10,0x10,0x1C,0x0D,0x00,0x00,0xFFF # GP=
S Time:2018/28/16-16:28:13.0
. . .

Post processing etc. all done after the fact since this was purely a loggin=
g application from my perspective.  Other programmers in the team dealt wit=
h sending the files up to the cloud and cleaning up folders.

For my new client my only additional task will be to create 'fake' CAN IDs =
with ERROR flag information.  The client application will need to parse the=
 message files looking for these perhaps other messages and then can back t=
rack as far as desired to determine what might have gone wrong.

That's if they decide to go in this direction.
John Dammeyer


> -----Original Message-----
> From: [email protected] [mailto:canlist-owner@vector-in=
formatik.com] On Behalf Of Kees Zagers
> Sent: November-03-20 2:25 AM
> To: [email protected]
> Subject: Re: [CANLIST] Logging CAN messages
>=20
> Hi John,
>=20
> Looks like a great product. The PIC32 is a nice controller for a CAN
> front end system. Mainly the flexibility in the CAN controller makes it
> very suitable.
>=20
> BTW I also added the multiple FIFO buffers (1024 messages) to the
> firmware, to use it as front end for an RPi. The prototype worked,
> unfortunately the customer did not get the order for production. For the
> interface to the Pi we used the I2C, which is better standardized than
> the SPI in my opinion. For the new CAN FD and upcoming XL however I2C
> will not be fast enough.
>=20
> Kees
>=20
>=20
> Op 1-11-2020 om 17:43 schreef John Dammeyer:
> > Thanks Kees,
> > I've passed on the information for your product to my client.  It looks=
 like I'm also going to be able to get one of these.  I've put a
> photo on my web site since I don't remember if attachments work with this=
 forum.
> > http://www.autoartisans.com/CAN/STU-2.jpg
> >
> > It's got a PIC32 front end and connects via SPI to a PiZeroW and has GP=
S and RTC.  Starts logging c/w RTC based time stamps on
> power up and once the Pi is alive and GPS TOD information is there it tra=
nsfers the files onto the Pi SD card.  Each file holds 4000
> messages which include TOD, Lat/Long/Altitude once per second.
> >
> > The only change to the PIC32 firmware will be info 'faked' CAN messages=
 reporting bus error conditions mixed in with the regular
> received CAN messages.  But that too will cost the customer so when I mee=
t with them next week I'll know more.
> >
> > John Dammeyer
> >
> >
> >
> >> -----Original Message-----
> >> From: [email protected] [mailto:canlist-owner@vector=
-informatik.com] On Behalf Of [email protected]
> >> Sent: November-01-20 1:12 AM
> >> To: [email protected]
> >> Subject: RE: [CANLIST] Logging CAN messages
> >>
> >> Hello John,
> >>
> >> Just one remark. I was not suggesting that you (or your customer) woul=
d
> >> make the change in the firmware. It would be an upgrade of the existin=
g
> >> software and done by myself.
> >>
> >> Kees
> >>
> >>
> >> John Dammeyer schreef op 31.10.2020 21:45:
> >>> Hi Kees,
> >>>
> >>> That product is close to what they need.  Nicely done!
> >>>
> >>> Although when I had the scope on their system last Thursday it looks
> >>> like the messages were running at least every 10mS.  I think they sai=
d
> >>> 100Hz, 50Hz and 20Hz intervals.  I'd say I'd have to have the ability
> >>> to store at least 5,000 messages at 500kbps.
> >>>
> >>> A few years ago I did do a similar logging project.  They wanted the
> >>> data stored on a Raspberry PiZeroW and then upload to the cloud via
> >>> WiFi when available.
> >>>
> >>> We found that no matter what we did the Pi still took about 12 second=
s
> >>> to boot.  Ultimately what I did was design and build a PIC32 front en=
d
> >>> to timestamp and queue all messages.  Once the Pi was awake the PIC32
> >>> became an SPI slave to the Pi which then pulled up the stored
> >>> messages.
> >>>
> >>> Unfortunately I don't have access to the hardware or legally the
> >>> software.  I was hoping that there might be something that uses a
> >>> KVaser or Port or some other USB dongle and a PC application that
> >>> would do this.
> >>>
> >>> I'll talk to the client and mention your product.  Not that intereste=
d
> >>> in making changes in Basic though.
> >>>
> >>> Thanks again
> >>>
> >>> John Dammeyer
> >>>
> >>> FROM: [email protected]
> >>> [mailto:[email protected]] ON BEHALF OF Kees Zagers
> >>> SENT: October-31-20 11:53 AM
> >>> TO: [email protected]
> >>> SUBJECT: Re: [CANLIST] Logging CAN messages
> >>>
> >>> Hi John,
> >>>
> >>> In principle I created this functionality in my SI2CBB
> >>> (www.unicanner.com [1]). The embedded hardware has all the CAN
> >>> functionality in this case. The CANLOG command I created has the even=
t
> >>> logging possibility. Button (connected to one of the Arduino pins),
> >>> specific message and error event can be defined by the user. The only
> >>> restriction is the fact that I used the standard circular FIFO buffer
> >>> of the built-in CAN controller of the Microchip PIC32. This buffer
> >>> only contains 32 messages. So the pre-trigger information is 32
> >>> messages, which probably will be a fraction of the messages during th=
e
> >>> last five seconds. The post-trigger information can be as long as you
> >>> want. The PIC32 has the possibility to create 32 of these FIFO buffer=
s
> >>> in RAM, where every FIFO can be linked to one or more filters and
> >>> masks. If all FIFO's are linked to the same filter and mask, they can
> >>> be cascaded, which gives us a buffer of 1024 messages. The problem
> >>> however in this case it is not circular; if the last FIFO is full it
> >>> stops. I have to check, but it should be possible to detect the buffe=
r
> >>> full interrupt of the last FIFO and return to the first FIFO again. I=
n
> >>> that case we can have at least 1 second of the pre-trigger messages i=
n
> >>> a normal CANbus system.
> >>>
> >>> Anyway it is difficult to say how large the circular buffer should be
> >>> to contain the information of 5 or 10 seconds, because it depends of
> >>> course on the bus speed and bus load.
> >>>
> >>> The result of the logging can be directly uploaded to a PC through th=
e
> >>> USB interface or locally stored on an SD flash card.
> >>>
> >>> So far my input.
> >>>
> >>> Cheers
> >>>
> >>> Kees
> >>>
> >>> Op 30-10-2020 om 06:50 schreef John Dammeyer:
> >>>
> >>>> I haven't kept up to date with newer developments over the last few
> >>>> years. My Lawicel CANUSB, RM CANUSB and monitor program plus
> >>>> whatever I've written has been adequate for the tasks I do.
> >>>>
> >>>> I've been asked to determine if a tool exists that can log CAN
> >>>> messages in a circular buffer. Say 10 seconds worth and when a
> >>>> trigger like either a message or say an error flag occurs the
> >>>> logging to the circular buffer goes for 5 seconds and then is
> >>>> stopped and saved.
> >>>>
> >>>> Essentially the system shows random faults and they'd like to know
> >>>> what happened for 5 seconds on either side of the event.
> >>>>
> >>>> The even can be: a contact closure like a button pressed by an
> >>>> operator; a specific message, Bus error as signaled by an error
> >>>> flag.
> >>>>
> >>>> Does a tool like that exist?
> >>>>
> >>>> Thanks
> >>>>
> >>>> John Dammeyer
> >>>>
> >>>> "ELS! Nothing else works as well for your Lathe"
> >>>>
> >>>> Automation Artisans Inc.
> >>>>
> >>>> www dot autoartisans dot com
> >>> --
> >>>
> >>> Kees Zagers
> >>>
> >>> SI-KWADRAAT B.V.
> >>>
> >>> Van Kemenadelaan 13
> >>>
> >>> 5674 PP  NUENEN
> >>>
> >>> Tel. +31(0)40-2839582
> >>>
> >>> [email protected]
> >>>
> >>>
> >>> Links:
> >>> ------
> >>> [1] http://www.unicanner.com
> >> --
> >> Archives and useful links: http://groups.yahoo.com/group/CANbus
> >> Subscribe and unsubscribe at www.vector.com/canlist/
> >> Report any problems to <[email protected]>
> > --
> > Archives and useful links: http://groups.yahoo.com/group/CANbus
> > Subscribe and unsubscribe at www.vector.com/canlist/
> > Report any problems to <[email protected]>
>=20
> --
> Archives and useful links: http://groups.yahoo.com/group/CANbus
> Subscribe and unsubscribe at www.vector.com/canlist/
> Report any problems to <[email protected]>

--
Archives and useful links: http://groups.yahoo.com/group/CANbus
Subscribe and unsubscribe at www.vector.com/canlist/
Report any problems to <[email protected]>