RE: Logging CAN messages

I'd have to dig into the firmware to see exactly what mode I used.  I know =
I was able to do 1Mbps although the target only used 500Kbps.

In this case I specified the hardware, someone else did the board design.  =
I used a Microchip Automotive Network Board with CLICK modules to wire to a=
 Pi3 for development.  I used the PIC32MX795F512L in the ANB but the end pr=
oduct used the PIC32MX795F512M and a PiZeroW.

Nothing like lots of horsepower for these types of projects.

Sequencing from power up was very important in that project so we used some=
 of the device bits in the 16 byte image for tracking.  Don't remember exac=
tly.  It was all 3 years ago.

John

> -----Original Message-----
> From: [email protected] [mailto:canlist-owner@vector-in=
formatik.com] On Behalf Of Kees Zagers
> Sent: November-03-20 11:51 AM
> To: [email protected]
> Subject: Re: [CANLIST] Logging CAN messages
>=20
> Hi John,
>=20
> Nice job. Fully agree with you that you have to do such a job in C and
> forget Python.
>=20
> Just one remark about the fake CAN message with the error flag. I guess
> it would be good to use the so-called "all message mode" of the PIC32
> controller. In that case you still have the information of the message
> as it was received (maybe the correct identifier and for sure the
> timestamp). Just add the error flag into the message on a bit which is
> not used. This is what I implemented in my CANLOG command.
>=20
> The PIC32 in my system (PIC32MX795) has 128 kB of RAM of which I can use
> only 32 kB maximum for buffering, because of the existing firmware. By
> using FIFO's of both CAN controllers and setting up the circular buffer
> over these FIFO's it would be possible to store a maximum of 2048
> messages. Maybe not enough, but it is a low cost solution.
>=20
> Cheers
>=20
> Kees
>=20
> Op 3-11-2020 om 18:43 schreef John Dammeyer:
> > Hi Kees,
> > To make this project work I used 10Mbps SPI.  The Pi was the master.  O=
riginally on the Pi side I started with Python and then
> threw it into the trash bin pretty quickly.  Hard to debug and slow as mo=
lasses.
> > Switching to C made all the difference.
> >
> > On the PIC32 side the system essentially recorded messages into a FIFO =
4096 records long with each record 16 bytes in size.  The
> CAN interrupt routine just read out four 32 bit words into the record and=
 incremented the queue pointer.  The compiled C code
> was examined at the assembler level to ensure that the code was as fast a=
s hand coding it in assembler.  Same with reading it from
> the FIFO.
> >
> > At the other end, the SPI data is also blocked into 16 byte records and=
 the hardware set up so the master can do a block transfer
> of 16 bytes at full speed.   So if a 500kpbs CAN messages takes about 250=
 uS on the bus then the transfer of the 16 bytes to the Pi
> takes 1.6uS done purely by the hardware.  An interrupt to the PIC32 when =
all 16 bytes were transferred.
> >
> > At the Pi end data written into files and a new file started after 4000=
 messages were received.  Three IDs were created for GPS
> information.  Here's a sample.
> >
> > STU Serial #000025  (Log #106)
> > Date: 2018-02-16 16:28:02
> > 01575000,16:28:02.893.75,0x18,0x7E,0x17,0x00,0x7F,0x61,0x00,0x00,0x03C
> > 01575001,16:28:02.898.75,0x19,0x7E,0x17,0x00,0xA9,0x60,0x00,0x00,0x03C
> > . . .
> > 01575188,16:28:03.830.25,0xE1,0x69,0x25,0x1F,0x84,0xA3,0xFC,0xFD,0xFFD =
# GPS Pos:  52.2545633,-3.3774716
> > 01575189,16:28:03.830.25,0xCA,0x07,0x00,0x00,0x00,0x00,0x00,0x00,0xFFE =
# GPS Alt: 199.4
> > 01575190,16:28:03.830.25,0x12,0x02,0x10,0x10,0x1C,0x0D,0x00,0x00,0xFFF =
# GPS Time:2018/28/16-16:28:13.0
> > . . .
> >
> > Post processing etc. all done after the fact since this was purely a lo=
gging application from my perspective.  Other programmers
> in the team dealt with sending the files up to the cloud and cleaning up =
folders.
> >
> > For my new client my only additional task will be to create 'fake' CAN =
IDs with ERROR flag information.  The client application will
> need to parse the message files looking for these perhaps other messages =
and then can back track as far as desired to determine
> what might have gone wrong.
> >
> > That's if they decide to go in this direction.
> > John Dammeyer
> >
> >
> >> -----Original Message-----
> >> From: [email protected] [mailto:canlist-owner@vector=
-informatik.com] On Behalf Of Kees Zagers
> >> Sent: November-03-20 2:25 AM
> >> To: [email protected]
> >> Subject: Re: [CANLIST] Logging CAN messages
> >>
> >> Hi John,
> >>
> >> Looks like a great product. The PIC32 is a nice controller for a CAN
> >> front end system. Mainly the flexibility in the CAN controller makes i=
t
> >> very suitable.
> >>
> >> BTW I also added the multiple FIFO buffers (1024 messages) to the
> >> firmware, to use it as front end for an RPi. The prototype worked,
> >> unfortunately the customer did not get the order for production. For t=
he
> >> interface to the Pi we used the I2C, which is better standardized than
> >> the SPI in my opinion. For the new CAN FD and upcoming XL however I2C
> >> will not be fast enough.
> >>
> >> Kees
> >>
> >>
> >> Op 1-11-2020 om 17:43 schreef John Dammeyer:
> >>> Thanks Kees,
> >>> I've passed on the information for your product to my client.  It loo=
ks like I'm also going to be able to get one of these.  I've put
> a
> >> photo on my web site since I don't remember if attachments work with t=
his forum.
> >>> http://www.autoartisans.com/CAN/STU-2.jpg
> >>>
> >>> It's got a PIC32 front end and connects via SPI to a PiZeroW and has =
GPS and RTC.  Starts logging c/w RTC based time stamps
> on
> >> power up and once the Pi is alive and GPS TOD information is there it =
transfers the files onto the Pi SD card.  Each file holds
> 4000
> >> messages which include TOD, Lat/Long/Altitude once per second.
> >>> The only change to the PIC32 firmware will be info 'faked' CAN messag=
es reporting bus error conditions mixed in with the
> regular
> >> received CAN messages.  But that too will cost the customer so when I =
meet with them next week I'll know more.
> >>> John Dammeyer
> >>>
> >>>
> >>>
> >>>> -----Original Message-----
> >>>> From: [email protected] [mailto:canlist-owner@vect=
or-informatik.com] On Behalf Of kees@si-
> kwadraat.nl
> >>>> Sent: November-01-20 1:12 AM
> >>>> To: [email protected]
> >>>> Subject: RE: [CANLIST] Logging CAN messages
> >>>>
> >>>> Hello John,
> >>>>
> >>>> Just one remark. I was not suggesting that you (or your customer) wo=
uld
> >>>> make the change in the firmware. It would be an upgrade of the exist=
ing
> >>>> software and done by myself.
> >>>>
> >>>> Kees
> >>>>
> >>>>
> >>>> John Dammeyer schreef op 31.10.2020 21:45:
> >>>>> Hi Kees,
> >>>>>
> >>>>> That product is close to what they need.  Nicely done!
> >>>>>
> >>>>> Although when I had the scope on their system last Thursday it look=
s
> >>>>> like the messages were running at least every 10mS.  I think they s=
aid
> >>>>> 100Hz, 50Hz and 20Hz intervals.  I'd say I'd have to have the abili=
ty
> >>>>> to store at least 5,000 messages at 500kbps.
> >>>>>
> >>>>> A few years ago I did do a similar logging project.  They wanted th=
e
> >>>>> data stored on a Raspberry PiZeroW and then upload to the cloud via
> >>>>> WiFi when available.
> >>>>>
> >>>>> We found that no matter what we did the Pi still took about 12 seco=
nds
> >>>>> to boot.  Ultimately what I did was design and build a PIC32 front =
end
> >>>>> to timestamp and queue all messages.  Once the Pi was awake the PIC=
32
> >>>>> became an SPI slave to the Pi which then pulled up the stored
> >>>>> messages.
> >>>>>
> >>>>> Unfortunately I don't have access to the hardware or legally the
> >>>>> software.  I was hoping that there might be something that uses a
> >>>>> KVaser or Port or some other USB dongle and a PC application that
> >>>>> would do this.
> >>>>>
> >>>>> I'll talk to the client and mention your product.  Not that interes=
ted
> >>>>> in making changes in Basic though.
> >>>>>
> >>>>> Thanks again
> >>>>>
> >>>>> John Dammeyer
> >>>>>
> >>>>> FROM: [email protected]
> >>>>> [mailto:[email protected]] ON BEHALF OF Kees Zage=
rs
> >>>>> SENT: October-31-20 11:53 AM
> >>>>> TO: [email protected]
> >>>>> SUBJECT: Re: [CANLIST] Logging CAN messages
> >>>>>
> >>>>> Hi John,
> >>>>>
> >>>>> In principle I created this functionality in my SI2CBB
> >>>>> (www.unicanner.com [1]). The embedded hardware has all the CAN
> >>>>> functionality in this case. The CANLOG command I created has the ev=
ent
> >>>>> logging possibility. Button (connected to one of the Arduino pins),
> >>>>> specific message and error event can be defined by the user. The on=
ly
> >>>>> restriction is the fact that I used the standard circular FIFO buff=
er
> >>>>> of the built-in CAN controller of the Microchip PIC32. This buffer
> >>>>> only contains 32 messages. So the pre-trigger information is 32
> >>>>> messages, which probably will be a fraction of the messages during =
the
> >>>>> last five seconds. The post-trigger information can be as long as y=
ou
> >>>>> want. The PIC32 has the possibility to create 32 of these FIFO buff=
ers
> >>>>> in RAM, where every FIFO can be linked to one or more filters and
> >>>>> masks. If all FIFO's are linked to the same filter and mask, they c=
an
> >>>>> be cascaded, which gives us a buffer of 1024 messages. The problem
> >>>>> however in this case it is not circular; if the last FIFO is full i=
t
> >>>>> stops. I have to check, but it should be possible to detect the buf=
fer
> >>>>> full interrupt of the last FIFO and return to the first FIFO again.=
 In
> >>>>> that case we can have at least 1 second of the pre-trigger messages=
 in
> >>>>> a normal CANbus system.
> >>>>>
> >>>>> Anyway it is difficult to say how large the circular buffer should =
be
> >>>>> to contain the information of 5 or 10 seconds, because it depends o=
f
> >>>>> course on the bus speed and bus load.
> >>>>>
> >>>>> The result of the logging can be directly uploaded to a PC through =
the
> >>>>> USB interface or locally stored on an SD flash card.
> >>>>>
> >>>>> So far my input.
> >>>>>
> >>>>> Cheers
> >>>>>
> >>>>> Kees
> >>>>>
> >>>>> Op 30-10-2020 om 06:50 schreef John Dammeyer:
> >>>>>
> >>>>>> I haven't kept up to date with newer developments over the last fe=
w
> >>>>>> years. My Lawicel CANUSB, RM CANUSB and monitor program plus
> >>>>>> whatever I've written has been adequate for the tasks I do.
> >>>>>>
> >>>>>> I've been asked to determine if a tool exists that can log CAN
> >>>>>> messages in a circular buffer. Say 10 seconds worth and when a
> >>>>>> trigger like either a message or say an error flag occurs the
> >>>>>> logging to the circular buffer goes for 5 seconds and then is
> >>>>>> stopped and saved.
> >>>>>>
> >>>>>> Essentially the system shows random faults and they'd like to know
> >>>>>> what happened for 5 seconds on either side of the event.
> >>>>>>
> >>>>>> The even can be: a contact closure like a button pressed by an
> >>>>>> operator; a specific message, Bus error as signaled by an error
> >>>>>> flag.
> >>>>>>
> >>>>>> Does a tool like that exist?
> >>>>>>
> >>>>>> Thanks
> >>>>>>
> >>>>>> John Dammeyer
> >>>>>>
> >>>>>> "ELS! Nothing else works as well for your Lathe"
> >>>>>>
> >>>>>> Automation Artisans Inc.
> >>>>>>
> >>>>>> www dot autoartisans dot com
> >>>>> --
> >>>>>
> >>>>> Kees Zagers
> >>>>>
> >>>>> SI-KWADRAAT B.V.
> >>>>>
> >>>>> Van Kemenadelaan 13
> >>>>>
> >>>>> 5674 PP  NUENEN
> >>>>>
> >>>>> Tel. +31(0)40-2839582
> >>>>>
> >>>>> [email protected]
> >>>>>
> >>>>>
> >>>>> Links:
> >>>>> ------
> >>>>> [1] http://www.unicanner.com
> >>>> --
> >>>> Archives and useful links: http://groups.yahoo.com/group/CANbus
> >>>> Subscribe and unsubscribe at www.vector.com/canlist/
> >>>> Report any problems to <[email protected]>
> >>> --
> >>> Archives and useful links: http://groups.yahoo.com/group/CANbus
> >>> Subscribe and unsubscribe at www.vector.com/canlist/
> >>> Report any problems to <[email protected]>
> >> --
> >> Archives and useful links: http://groups.yahoo.com/group/CANbus
> >> Subscribe and unsubscribe at www.vector.com/canlist/
> >> Report any problems to <[email protected]>
> > --
> > Archives and useful links: http://groups.yahoo.com/group/CANbus
> > Subscribe and unsubscribe at www.vector.com/canlist/
> > Report any problems to <[email protected]>
>=20
> --
> Archives and useful links: http://groups.yahoo.com/group/CANbus
> Subscribe and unsubscribe at www.vector.com/canlist/
> Report any problems to <[email protected]>

--
Archives and useful links: http://groups.yahoo.com/group/CANbus
Subscribe and unsubscribe at www.vector.com/canlist/
Report any problems to <[email protected]>