SD/mmc card datalogger storage, non FAT, make the picaxe think its eeprom

G

Gwillo

New Member
Hi, i am looking at embarking on a datalogging project based around the rev-ed product, but probably changing the chip for one with more inputs, changing the code, but most importantly changing the data storage as even 8 eeprom's wont be enough.

I know a lot of people on here have been experimenting with FAT based solutions to make it faster to download the logged data, however I am not particularly worried about this and would rather read it back through the chip and using plug to hook it up to the wizard in the rev-ed picaxe software.

Has anyone been able to do this, as it sounds like from the posts that this is possible but most threads are focusing on implementing FAT. If not can anyone suggest a sensible medium that i can easily store the data to and read back through the chip as if it were eeprom. Basically id love just a huge eeprom, or some way i can trick the picaxe into thinking the medium is an eeprom.

Sorry if my questions sound silly but i have to admit im a little out of my depth in some of the code im seeing in threads on this site, I have used PICAXE before but nothing on this scale, hense why id like to try and simply modify an existing solution where i can use a wizard, or modify the code the wizard generates.

Many Thanks :)
 
B

BeanieBots

Moderator
Welcome to the forum.

How much data?
You are only limited to 8 EEPROMs by the chip address lines.
A 3 to 8 encoder chip on the enable lines will allow 8*8 for just three IO pins.
No reason you can't go even higher.

What about how long it will take to read/send serially all that data back to the PC?
 
M

MFB

Senior Member
DataFlash again!

It is possible to store a maximum of 512Kbytes using four 24FC1025 EEPROM chips and standard PICAXE I2C commands. However, what I would really like to see is code for using Atmel DataFlash. These SPI devices can store Mbytes in a single 8-pin package.
 
G

Gwillo

New Member
Thanks guys for the quick reply.

As i said i want to try to keep the system as simple (and as close to a configuration that i know works) as possible. The transfer time isnt an issue, the plan is to take about 8 or 9 sensor readings at either 1 or 2 or 4 times a second for a little over 90 minutes. I will have hours to load this onto a laptop if required, I just want something that i know is fairly reliable even if it is a bit of an oil burner.

I like the sound of the encoder chip, as i think we could probably get enough memory using 8 arrays of 8 512 chips, would this be hard to do as far as the code is concerned and does it complicate the read back when transfering back to the computer.

I have had a quick look at the flash device that was recomended, the larger ones do however seem a tad "beta" according to Amtels own website :
http://www.atmel.com/dyn/products/product_card.asp?part_id=3815
"Not recommended for new designs"

Also i dont know how the flash and spi would behave with the wizard that i was hoping to use to generate most of the code and then modify, so the flash idea is restriced not by its abilities but by mine.

Once again thanks.
 
B

BCJKiwi

Senior Member
While the discussions on SD/MMC with FAT usually relate to storage, these devices can be read as well.
So not only can the data be written and read by the PICAXE it can also be written and read by the PC (provided consistent data layouts are used within the file).
 
Last edited:
B

BeanieBots

Moderator
I make that around 216k bytes max. of data assuming single byte readings.
MFB's solution gives you 512k.

However, if you want to use 512 chips, then you will need some sort of encoder. How you do it depends on how many pins you have spare.
You could use one pin for each bank or you could use an encoder chip.

Reading and writing would only require keeping track of which bank is enabled.
Thin kof it as an extension to the address value in the I2C command.

As for "looks a bit beta".
"Not recommended for new designs" usually means "going out of prodcution".
ie no longer supported as apposed to under development.

SD and FAT does of course have the great advantages pointed out by BCJKiwi.
 
G

Gwillo

New Member
Has anyone actually managed to get an sd card talking to a picaxe reliably yet, because all of the posts that i have come across seem to be a wash with problems of trying to get the FAT file system to work.

The only instances of this that i have seen working is the systems using some sort of serial to sd card module that all seem to be in the region of $25-30 + delivery (To UK).

If possible id like to stay clear of FAT as it seems to cause a lot of headaches, and again the programming would almost certainly be beyond me unless someone else already has code which does this and would just need modifying.

Just to clarify the readings wont be simple on/off readings, they willl be digital temperature sensors mainly so 3 digit reading.

I had a look at the data sheet for the 18x datalogger from rev-ed and is says that 8 x 24LC256 is good for 32768 readings, so 8 x 24lc512 would do for 65536 if my maths is correct, for 8 readings at 2hz for 90 mins, 8x120x90=86400.

Yes BeanieBots having had a closer look that the flash your coment about me thinking it might be a little beta is probably correct as there are data sheets there from a number of years back. My bad:)

As for the encoder stuff, is the encoder a dedicated chip or is it simply another picaxe comunicating via serial data, and is it easier to use a separate encoder or is it easier to do it all on the chip,(im guessing this would depend on how many arrays i needed)

Thanks
 
D

demonicpicaxeguy

Senior Member
Gwillo said:
Has anyone actually managed to get an sd card talking to a picaxe reliably yet, because all of the posts that i have come across seem to be a wash with problems of trying to get the FAT file system to work.

The only instances of this that i have seen working is the systems using some sort of serial to sd card module that all seem to be in the region of $25-30 + delivery (To UK).
Click to expand...
yes and no, i have gotten it to work with a pic18f4550 and a pic16f88 with an sd card so far reading a writing very reliably mmc cards and getting very very rare and these days aren't worth making the firmware compatible for

of all the spi devices to use the sd card is pretty bad and difficult to interface to you can look at a dozen different datasheets and get 2 dozen different circuits for them,

if you build a circuit for a card make sure you have a "clean" power supply,
regulated between 2.8v and 3.6v , put a pullup resistor (10k to 100k) on the dataout line and keep your lines between the picaxe and sdcard as short as possible, you may also find in some cases a smal capacitor is needed between gnd and v+ close to the card as well

and keep your mobile phone at least 1 foot away (don't ask i learnt this the hard way)

one easy way to get arround the fat file system implementing problem is to read and write to the card sector by sector and simply read the raw data off the card instead of through a file system, there are a few vb6 examples of doing this,
 
B

BCJKiwi

Senior Member
@ Gwillo

I can understand your being wary of FAT.

Tried a few options all unsuccesful until I discovered the uAlfat device range.
Have (since Jan 2008) had successful operation of the uAlfat-TF (TF = Trans Flash = Micro SD) running on hi2c. See sample code here: http://www.picaxeforum.co.uk/showthread.php?t=8513

Hence the comments in my earlier post #5
 
Last edited:
B

BeanieBots

Moderator
Gwillo, be careful not to get your bytes, bits and digits mixed up.
You need to be VERY specific about the data you want to record.
Also, record what actually comes back from the sensor. DO NOT record the data after it has been processed for any reasons of display or units conversion.
Don't forget 255 is a three digit number, but it's just one byte of data.
If you need more, 65535 is a five digit number but only two bytes.
This was all covered in greater depth just a few days ago so I'll not repeat it all here.

The memory bank encoding is dead easy.
The EEPROMs have an enable pin. Common up all the enables on each bank of eight. Then either, connect each set to a PICAXE output or (to save on outputs) use a 3-to-8 encoder chip or any other logic you fancy to do the encoding. New commands have recently been added which can do the encoding for you but it's just as easy to use high/low commands for this app.

Don't forget that EEPROM has page breaks. As you are not recording data very fast, I'd avoid the problem completely by slapping a pause 20 after every byte write.
 
G

Gwillo

New Member
does anyone have a circuit diagram for any of the uAlfat boards or know where i can get a data sheet and circuit diagram for one?

I would prefer if possible to build anything like this into my own pcb so would rather not use preassembled boards if possible, the reason i ask is that i have spotted this

http://www.crownhill.co.uk/product.php?prod=1578

however it is surface mount and pins are very close together so a home made pcb might not be up to it as the tracks would need to be very small, i might just be better going with this one to save me the almost enevitable hassel:

http://www.crownhill.co.uk/product.php?prod=1785

Thanks for sharing this with us BCJKiwi, i had read your thread previously but wasnt so sure if would be suitable for my aplication however reading it again it looks like it could be ideal, obviously the device uses FAT, so can the device be read back through the chip to the serial connection to the computer for reading still, or does it have to be plugged into a computer. I'v got plenty of TF (microSD) cards liking around from older mobile phones, ill just have to try not to put the SDHC version in with 4gb because it probably wont like it.:)

As for demonicpicaxeguy's spi solution interfacing with the SD, the mobile phone interference issue may well be an issue as the plan for the future is to add a radio modem to the system to allow us to read the data being recorded roughly in real time. So there may be an issue there.

I am planning on using either a 28 or 40 pin chip so hopefully the number of inputs and outputs shouldnt be to much of a problem with the idea of using banks of chips, i dont know how many figures the digital temperature sensors go to i think its only three, however one of the other possible imputs i am looking at is a GPS chip, which obviously would be more and if my memory serves me right picaxes dont like numbers bigger that 65535, BeanieBots, you couldnt possibly post the link to that previous thread you were talking about with the bits and bytes just to make sure i dont make any schoolboy errors, its been a couple of years since i did most of this stuff.
As for enabling the banks of chips, am i correct in thinking this is simply putting the Write-Protect PIN (7) high or low to select the bank currently in use?

I have to say it would seem that the most native solution is the banks of EEPROM, i am uneasy about buying modules which are reasonably expensive as i am currently able to get components for next to nothing as a uk supplier is involved with the project which the device is intended to be used on, but i wont go into details about that, just that it makes sense for me to try and make something using lots of basic components than a few rather exotic parts, 1) financially and 2) because the programming is probably easier

all im trying to say is that i dont mind having a ridiculous number of EEPROM's attached to this thing, and i think ill let someone else sit by the computer while the data loads up :)
 
D

Dippy

Moderator
"As for demonicpicaxeguy's spi solution interfacing with the SD, the mobile phone interference issue may well be an issue ..."

- can you expand on that. I don't understand.

Some Sd card holders are all-metal and they use SD cards in mobile phones so obv not a problem for a good designer who has heard of copper planes.

I ask this as I'm currently doing an SPId SD controller with UART interface.
Not so good as UallFAT but with some extra bits that UalFAT can't do.

And, if I were you, I'd buy the ready-built (2nd link). For the sake of a few quid you will kick yourself if A) you screw up the chip, and B) You spend days/weeks trying to get a home-brew PC to work.

Have no doubt, the fastest solution will be EEPROM or FRAM. And wth FRAM especially you can have continuous fast storing limited by addresses and your wallet. The downside is that with FAT on SD you can just pull the card out and stuff it any PC. Extracting oodles of data and uploading to your PC (poss. requiring some extra software) via PICAXE will be a slow business - esp if you do error checking.
If doing this on-site the advantage of swapping an SD card versus faffing around uploading is obvious - and its bound to rain.

Of course once we get continuos writes onto high speed SD (video style) then no worries.
Actually thats got me thinking...
 
B

BeanieBots

Moderator
Gwillo, here's the link to the other thread.
http://www.picaxeforum.co.uk/showthread.php?t=12904

As for enabling banks of EEPROM, I own you an apology.
I2C EEPROM does NOT have a chip select (enable) pin. The write protect would only work when writing. You would get a data clash when reading.

It is SPI EEPROM which has chip select. Unfortunately, SPI EEPROM does not have address lines.
So, your options are:- (excluding SD/FAT)

Up to eight I2C EEPROM direct on I2C bus.
Multiplex the I2C bus using a 4066 quad switch or similar.
This could be done in banks of eight EEPROMS.

Multiple SPI EEPROMs using chip select from a decoder.
An 8-bit IO port would allow 256 chips to be used but basic encoders only come with 8 or 16 outputs. Depends how much you NEED.

There is also another type of EEPROM which is supported by the X2 type PICAXE. These are uni/o one-wire devices.
A single 11LC160 will hold 2048 X 8 bits of data.
http://ww1.microchip.com/downloads/en/DeviceDoc/22067E.pdf
Only one IO pin is required to use them.
Again, you could multiplex with a 4066 or similar.

I strongly suggest that you work how much data you actually NEED to store before you decide on which or how much memory to use.
Also, ask yourself very seriously how often it is even valid to take measurements. I don't know what you are measuring so cannot comment but measuring temperature every 0.25S has to be questioned. Very few sensors can even react that quickly.

Lots of descisions to make but many will be decided for you once you have properly analysed the real requirements of the project.

Oh, thanks Dippy, forgot about FRAM.
http://www.ramtron.com/files/datasheets/FM25L256Bds_r3.0.pdf

32768 X 8 using I2C.
That one DOES have chip select, so eight would be easy.
256 of them certainly possible. (as Dippy says, limited only by your wallet).
 
Last edited:
G

Gwillo

New Member
I guess the gps/radio modem interference problem with the sd card should be easilly solved sheilding them from each other, as for my comment on interfernec with the radio modem, it was based on what demonicpicaxeguy said about problems he had had with his mobile phone interfering when using spi, as with you i would have thought this wouldnt have been a problem as t-f cards are used in mobiles but presumably they are sheilded in the phone. I'd ask demonicpicaxeguy if i were you because im not entirely sure what waves cause what interference with exactly what. Its also a little beyond me, but if a good old bit of tin foil solves the problem then there we go, ill probabaly need to sheild all of these electronics as they are likely to be installed right next to a huuuuuuge motor.

I'm probably going to go down the EEPROM route for now as it is something i can build up from a simple 18x data logger by one by one changing components from a configuration i know works. I can do the upload inside so hopefully the rain isnt a problem, then again that dodgy roof...

BeanieBots, im going to go and work it out now, the temperature sensors dont need to be read as often as i orignally said, i do need some of the sensors measured very often (eg GPS, and a hall effect sensor) but as you say temperature doesnt really change that fast so i could probably surfice doing this once every say 10 seconds, as the reading from these arent critical for logging they are just being recorded for safety reasons and to stop us overheating things as warning devices are going to be fitted. I was worried that logging different things at different frequencys would possibly mess up the data logging as would 18x simply log no characters in that position in the log, this is where my knowledge on this starts to get a little patchy. If it is a problem would i possibly be better just to use two loggers.

Off to go work out what storage i need
 
G

Gwillo

New Member
anyone know how many bytes are needed to store a gps position, i dont know how the picaxe would handle it.
 
B

BeanieBots

Moderator
It will log what YOU tell it to log. YOU are in charge.
It not a commercial datalogger which does what somebody else decided.
It is YOUR datalogger. It logs EXACTLY what YOU tell it to log.

So, in addition to working out how often, you also need to design a memory map. (don't get scared, it's easy).
It's as easy as:-

Location 0 Temperature
Location 1 Humidity 1
Location 2 Humidity 2
Location 3 Voltage

Then repeat or whatever for whatever sensors you have.
It only gets complicated if you want to do things like change how often it updates depending on a reading like temp every minute but change to every 10 seconds if it gets hot or some other condition.

Don't forget, the map is important, because you need to read it back and get things in the right order.

You can also save masses of storage by only recording changes but lets only worry about that if you find you REALLY do have a problem getting enough EEPROM.
 
Last edited:
G

Gwillo

New Member
ah that sounds simple enough using the map, i was worried that if i missed out records it might move all the data around but now i understand.

Without the GPS position, which i want to record at 4Hz for 100 mins, i need 52800 bytes

I think... :)
 
D

Dippy

Moderator
4 samples per second for a 100 mins x Number_of_Data_Records_per_sample x Number_of_bytes_per_Data_record isn't it?
 
G

Gwillo

New Member
Just to clarify here is what i am planning to monitor:

Speed 1byte 4Hz 24000bytes
Hall Effect 1byte 4Hz 24000bytes
GPS ?????? 4Hz
Voltage 1 1byte 0.2Hz 1200bytes
Voltage 1 1byte 0.2Hz 1200bytes
Tepmperature 1 1byte 0.1Hz 600bytes
Tepmperature 2 1byte 0.1Hz 600bytes
Tepmperature 3 1byte 0.1Hz 600bytes
Tepmperature 4 1byte 0.1Hz 600bytes

52800bytes + gps data

I dont know how many bytes are needed for a gps position to be logged, if anyone knows i would really apreciate it.:)
 
B

BeanieBots

Moderator
Yes, it is that simple Dippy, except Gwillo is keeping the format of his raw data really secret so we can't help beyond that.

Maybe he has one of those military GPS systems that can resolve down to 10cm.
Maybe his hall effect is being read by a fancy 16-bit DVM which outputs as 5 digit ASCII on a serial port.

EDIT:
Good timing, crossed with your post Gwillo.
For the GPS stuff, you need to convert to a convienient scale and resolution for longitude and latitude. So, mimimum would be two bytes per reading but that would only give a resolution to the nearest 97 miles, so I guess you want a bit more.

A word variable could resolve down to about 600 meters.

I take it you REALLY DO have one of those military ones if you are reading every 0.25S.
Either that, or you are moving rather fast.
A 'typical' GPS is good for about +/- 50m. To move enough to see a difference in 0.25S would mean speeds around 400 meter per second!

EDIT2:
Don't forget to read this thread which is going on at the moment.
http://www.picaxeforum.co.uk/showthread.php?t=12960
 
Last edited:
G

Gwillo

New Member
I had thought of possibly using the GPS system that is featured here:

http://194.201.138.187/epages/Store.storefront/?ObjectPath=/Shops/Store.TechSupplies/Products/GPS001

the GPS is something that i would look to do at a later date to be honest, the whole datalogger is going to hopefully be fitted to a moving electric vehicle which will reach speeds in the region of 35mph, as earlier stated i want to use a radio modem with the serial out from the PICAXE to give me basic telemetry. The data modem will be a later addition to the system and the GPS is really for monitoring where the vehicle is now rather than where it was afterwards looking at the log.

As for the hall efect sensor we only need the reading to be accurate to an amp, the adc readings from it are a facter of five greater than the current passing through it. eg a reading of 125 - 25 amps.

The whole thing will have a 70 amp circuit breaker fitted and should usually opperate around 26 amps so the corresponding adc reading should only require 1 byte. So i think a 1 byte data log for the hall effect sensor should be adequate, as for the gps i dont know exactly how they transfer the data of thier position via serial, so hense why i dont know how much space will be required to store the position.

EDIT
I was under the impresion that most GPS modules were accurate to 10m, but i may well be wrong, just for the record here is the unit i would probably end up using http://www.rfsolutions.co.uk/acatalog/Board_Level_GPS_Receiver_Module.html
or this one:
http://www.rfsolutions.co.uk/acatalog/GPS_Module.html

EDIT2
Looks as though this is what iv been looking for on the data sheet for the first link

"NMEA Messages
The serial interface protocol is based on the National Marine Electronics Association’s NMEA 0183 ASCII
interface specification. This standard is fully define in “NMEA 0183, Version 3.01” The standard may be
obtained from NMEA, www.nmea.org"

"GLL - LATITUDE AND LONGITUDE, WITH TIME OF POSITION FIX AND STATUS
Latitude and longitude of current position, time, and status.
Format:
$GPGLL,<1>,<2>,<3>,<4>,<5>,<6>,<7>*<8><CR><LF>
Example:
$GPGLL,2447.2073,N,12100.5022,E,104548.04,A,A*65<CR><LF>"

"Field Example Description
1 2447.2073 Latitude in ddmm.mmmm format
Leading zeros transmitted
2 N Latitude hemisphere indicator, &#8216;N&#8217; = North, &#8216;S&#8217; = South
3 12100.5022 Longitude in dddmm.mmmm format
Leading zeros transmitted
4 E Longitude hemisphere indicator, 'E' = East, 'W' = West
5 104548.04 UTC time in hhmmss.ss format, 000000.00 ~ 235959.99
6 A Status, &#8216;A&#8217; = valid position, &#8216;V&#8217; = navigation receiver warning
7 A Mode indicator
&#8216;N&#8217; = Data invalid
&#8216;A&#8217; = Autonomous
&#8216;D&#8217; = Differential
&#8216;E&#8217; = Estimated
8 65 Checksum"
SOURCE: http://www.rfsolutions.co.uk/acatalog/DS320-2.pdf
 
Last edited:
B

BeanieBots

Moderator
Well it's all there, have a read.
http://www.rev-ed.co.uk/docs/GPS001.pdf

Note:
position to 5m. (not bad!)
Refresh rate 1Hz. (so no point trying to read at 4Hz).

GPS data comes out as a string. You don't want to go storing that.
For absolute position accuracy anywhere on the planet, see the figures in my previous post. That's based on circumference of the earth and the resolution you use. You could smaller data size by logging RELATIVE postions but it wouldn't work if you moved it far from wherever you programmed it to work.
 
D

Dippy

Moderator
There are a zillion different GPS modules out there with many different specs and behaviours.
Why not read the Data Sheet of the on you are interested in?

Example (picked at random)
http://www.rfsolutions.co.uk/acatalog/DS320-2.pdf

and it gives the data format too - you should be able to work out how many bytes you need to read and store.
Read and store - that's what you need to do too :)

I'm sure if you search this Forum there has been tons on reading various GPS modules.
Spend 20 minutes doing a search - or wait for ecloogle to log in ;)
 
G

Gwillo

New Member
Thanks guys for your help. Im going to have a look through the things you've suggested on the weekend as the rest of this week is pretty hectic for me, but thanks for all your help. Ill probably be getting cracking with actually building the thing in september so ill keep you all posted.

The GPS stuff seems a little intensive on me and the storage so i might avoid that for now but i will definately have a look later at the other threads:).
 
G

Gwillo

New Member
so its looking like 8 bytes per reading on the GPS, if its that many maybe a 1Hz record would surfice, perhapse using the real time value if i ever get around to putting an on board LCD onto this.

So the GPS will need 8x60x100=48000bytes, giving the information that i want

so overall im looking at 100800 bytes so i should be able to get enough using a basic EEPROM setup then?

EDIT: hang on iv got my bits and bytes messed up as you warned
 
Last edited:
G

Gwillo

New Member
BeanieBots, sorry i have only read some of the information. I may have misunderstood it but it seemed that using the link you gave i would need 8 bytes per reading, i would want it all logged on the same logger so i would want to take the serial out from the GPS and put that into the picaxe, and then log that. Having had a closer look yes your right 1Hz is the refresh rate on the unit. So this will have to be good enough so i will want to record this at 1Hz.

It is highly possible, being a bit of a newbie, that i have completely misunderstood the whole document however, so i might be trying to do something which seems quite silly to those in the know, unlike to an amature such as myself.

Thanks guys. I think ill probably have to wait and see which GPS module i end up with as that will obvously change a few things. Bring on September!
 
B

BeanieBots

Moderator
Gwillo said:
so i would want to take the serial out from the GPS and put that into the picaxe, and then log that
Click to expand...
Are you one those people who goes to a supermarket, packs it all into bags in the trolley, unpacks it onto the conveyor, packs it back into the bags, to take to the car, removes it from the bags to pack into the boot, packs it back into bags from the boot to take to the house. Unpacks from the bags onto the shelf, then packs it into the appropriate cupboards?

WHY?
The PICAXE can simply read the already logged data when you send to the PC.
If you were using SD and removing it to put in a PC it might make sense but you've already said you dont wont to do that.
I'm sure you have a valid reason.
 
You must log in or register to reply here.
Top