Datalogger design memory issues

[CliveWilcox]

I have the following requirement for logging inputs & outputs. The project operates underwater so it has to be small & by small battery. I can make several boards and spread them around if it helps!

It is a fixed underwater ROV (except it is vision blind)

The data needs to be displayed on LCD 16x2 for testing - human calibration. Or if anyone know how to wire up a better display let me know. I will also need to store collected data in memory for download later.

Logging time. If sample duration is for 3 hours the i want everything to shut down after 4 ( without data loss) Like wise, the mission could last as long as 10hrs so shutdown would be after 11hrs. ( It depends on its mission and required safety margin). I guess this requires battery backup

Problem 1. The devices for RTC have good points and bad points. Devices i have looked at so far are DS2417 - This is a lapsed timer. Would this be better than a RTC like DS1307. I notice that you can also get I2C devices with trickle charge. That would solve battery backups. The issue is that having read the data sheets I don't really understand them (new technology) can i date stamped and count the duration of the mission with one RTC device or do i need the lapsed timer as well? The mission is set active by a switch on the surface but only starts logging under water.


Parts so far - please suggest other options if you think they would be better.
1 x 28x1 or 28x2 - Have not decided yet.
2 x Honeywell HIH400-001 Humidity sensor - I understand and have these working
2 x Dallas DS18B20 Temp sensor - I understand and have these working
1 x Gas sensor - Outputs Inverted RS232 data - I understand and have these working thanks to another project and help from the forum. thanks guys. Gives 4 readings - but only two are used. Sample rate is set to 1sec, but it could be reduced to once evry 10secs at a push.
1 x fan complete with encoder (encoder is used on count input & shows only that it is working - speed not important)
1 x pressure sensor upto 14bar - I understand and have these working

Next question. How do i work out how much memory i need to log someting like this. I will be happy to have a flag set on an event if it helps - like fan stops, encoder output flaged error - time stamped. Alram LED light.

The motor fan is driven by PWM to save battery power.

There is also a few other LED and LCD alarms when certain conditions are passed, like low battery...

If any one can give a bit of advice mainly with memory storag that would be great. I must confess memory and data storage are my real muppet moments. It just how ever much i read data sheets on picaxe sheets it goes in omne brain cell and out the otherside. I guess i can't be good a everthing. Give me a block of metal and i can make dreams come true. Sigh

 

[eclectic]

@Clive.

Coo, a seriously involved project.

No answers I'm afraid, just more questions:

"Next question. How do i work out how much memory i need to log someting like this. I will be happy to have a flag set on an event if it helps - like fan stops, encoder output flaged error - time stamped. Alram LED light."

What needs to be logged? / Format? / Frequency? / Total time?

And another thought.
Two Axe110 dataloggers might do the task.

It could be worth skimming the datasheet and the PE wizards.


e

 

[BeanieBots]

Lots of questions but not much for us to go on.

Shutting down.
Don't understand why you want battery backup. Data can be stored without the need for power.

Storing data.
Data is stored as bytes. Two bytes are required for a word.
A number between 0 and 255 can be stored in one byte.
A number between 0 and 65535 can be stored in two bytes.
A 'typical' EEPROM can store 32768 bytes. (you could have 8 of them)
What is the numeric range of what you wish to store?

Do you need to store the date and time with every reading or just a start date/time and then a reading every 10 seconds or something else?

How accurate does the timing need to be?
Could be possible without any timing chips depending on requirement.
DS1307 would give you REAL time. No need for any extra timers.
28X1/2 perfectly capable of doing lapsed time themselves but not as accurate long term as DS1307.

 

[CliveWilcox]

eclectic

Data to be logged.

2 humidity as **.** = let w1 = w1 - 40 * 100 / 157 ‘ change to %RH... as per Rev site
2 Temp as **.** - or is it just **! - readtemp12 w6
1 gas mix ***** inverted serial "A *****"
1 gas mix ***** inverted serial "B *****"
These need to be date stamped.

Motor encoder fail - flag b1 = 1 and time stamp.

Mission start time Logged Full date stamp DD/MM/YY HH:MM:SS
Duration of mission b2 = "could be any duration in hrs or maybe mins"
Cut of mission b3 = " 1hr or 60mins - depends on above"

Ideal sample rate would be evrey 1 Sec, if memory allows. If not 1 in 10sec sample rate will have to do. It really depends on how much memory it takes up & how much i can get it the unit.

I intent to use free software to use on the PC to download the data. Something like stampplotV.


I have a AXE110 - thats where the idea came from. Mission software limited but a good guide.

BeanieBots

Thanks i just assumed that if every thing shuts down you need a battery to back up the memory. (My CNC dies if i turn it off if the battery fails - must be different kind of memory then.) Guess if it is shut as part of the program then i don't need it, assuming chips retain data.

I guess a start time would suffice. A little routine in the download software to increment the stamp would work.
With regard to the timing precision. It does not have to be that slick. About 1 or 10 sec is close enough sample rate.

The DS1307 according to the data sheet can only lapse time to 99.99mins

Interesting -I could use either 28x1 or 28x2 since they have the same pinouts.

 

[lbenson]

Data storage requirements: for instance, if you want to read your DS18B20 every 10 seconds for 11 hours, and you use the readtemp command, which gives as its output a single byte, then you would need 11 hours times 60 minutes times 60 seconds divided by 10 seconds per reading times one byte or 3,960 bytes. Twice that if you use readtemp12.

If you log the starting time and use timing within your program so that you know that you are logging a reading every 10 seconds, then you don't need to log the time with each reading (if the approximate timing thus attained is sufficient).

Similarly calculate the amount of logging storage needed for each sensor for the longest period of time required, and add them all up.

The 24Cxxxx-series I2C eeproms may give you your easiest memory solution. 8-pin dips up to at least 24C1024 (128K bytes) are available (I'm not sure if there are picaxe addressing issues with using the larger ones--if there are, you can gang them as BB suggests, and select using separate addressing lines).

For time stamps, I like the format YYMMDDhhmmss--then everything can be sorted easily.

 

[BeanieBots]

The DS1307 according to the data sheet can only lapse time to 99.99mins

Really! What about the days, months, and date registers.
You could always keep a record of the months in a byte variable.
That would give you around 21 years. (also handles leap years).

Alterantively, you could use the SQW output (at 1Hz) and use it to trigger an interrupt. Count up to 65535 in a word. That's about 18 hours worth.

You might want to reconsider your format for saving.
Save the RAW data and process later in excel or whatever.
Then it's just a question of saving byte or word values.

 

[eclectic]

@Clive.

Another possible (very large) storage option
from Dr Acula:
http://www.picaxeforum.co.uk/showthread.php?t=11181&highlight=08M+megabyte*

e

 

[BCJKiwi]

Since time accuracy is not overly critical, the same model could be adopted that I have used on a project.

Use an RTC.
Read each sensor in turn, storing results into specific consecutive scratchpad locations,
Read the clock and store its results.
A single write routine then writes the whole data set including a single time stamp to uAlfat (but could be eeprom).
Alarms could read/write specific records as and when required including a time stamp.

Have this running in a 28X1 @16MHz. It completes reads of 5 inputs, outputs display info to 4 x MCP23017s and writes to the uAlfat.
It achieves this at a rate of over 10 per second (over 2000 bytes of code in the main loop).
In this instance, as suggested, count the sqw from the clock to set the time to run each dataset.

Settings/configuration are carried out using an IR remote

There is info on part of this project here:- http://www.picaxeforum.co.uk/showthread.php?t=8513&highlight=Ualfat

 

[CliveWilcox]

lbenson

Thanks - the mist are starting to clear,

so for this example you give = 3906bytes, this would fit into a 4k space ie 4096bytes. Going to need a big memory chip i think. I assume i will get best data storage if the data is stored in 8byte chunks.

beaniebots

My error - just re-read the data sheets on a DSI388 (one option) and i had mis-read the detail. It was the watchdog timer that can only count upto 99.99 (secs). Sorry to miss guide everyone.

SQW output is an option. I have never looked at that type of solution. I will look at it closer. The datsheet appears to indicate that it goes in steps, so i would have to use a combination of steps to get to the closest time frame i need. Yes?

Output will have to be information as required in serial. ie

A=temp1, B=temp2, C=RHD1, D=RHD2, E=Fan 0/1 (flag) count "****"

Start logging:

Date stamp start & count

YYMMDD HHMMSS *** (log duration) ** (add margin) ,(as per prior suggestion)

Main Log in 10sec steps


A **.** B **.** C **.** D**.** E *

write to memory & send to display

End logging

Shut down

I have to output in a way that the LCD can use the data. It makes sense to store the same, as it may need to be read by other devices. Not just for downloading the log. It is a real time monitor with log. This display part in mind, would i be better off using an additional picaxe to handle the screen display (still don't like 16x2's) and store data raw?

eclectic

My neck of the woods.

Just looked at the link - wow thats some storage

BCJ

I think that this is the way i am looking after spending most of yesterday and today looking at this problem.

Nice link to some very nice toys.

Alas i just do my own stuff since i need to make quite a few of these for this project I'm doing. Also since is just for me to satisfy my stupid ideas it keeps the costs down. I may make some money to pay for it with a few spin offs. I never have yet, but this is the biggest multi project to date. I hate to imagine what will happen if i start on planes or rockets. At least with water projects i can jump in and swim to get my bits back.

 

[BeanieBots]

Clive,
No it doesn't make any sense at all to store the data in a way that it needs to be sent or interpreted.

Let's say you have just read a DS18B20 temperture sensor in 12 bit mode to get the best possible resolution.
Let's say the reading came back as 404.

How would you display that?
Maybe as 25.2 degrees C.
It would require four bytes to store "25.2".
It would only require two bytes to store 404, so with just that one reading, you have halved the amount you could store. It would also take longer to convert and store but that might not be an issue.

There's another gotcha waiting to happen when you store 'converted' data.
Keeping with same example. Let's convert 25.4 C back to raw data.
It comes out as 403. Oh dear, you've lost some accuracy as well as storage space.

If you want to display in 'real' units. Store the RAW data and convert when you display. You have to the conversion at some point, it's just a question of doing it after storage rather than before.

It might even be worth considering storing in a 'special' format just for the purpose of storage.
If for example, you know your temperature will NEVER go outside the range 20C to 35C, you could subtract 320 (20 degrees) from the raw data and save all temperature readings as just ONE byte.

Using the SQW OP is simple.
Just use it to trigger an interrupt every second.
The interrupt routine can handle counting the seconds.

Also, be aware that the DS18B20 sensor can take up to 750mS to return a value when using the built-in commands. The method I always use when logging at 1 second intervals is to use the one wire commands. You end up with a one second lag on your data but it gives you an extra 700mS of processing time between reads. 750mS is a lot to lose when you've only got 1000mS to play with!

EDIT:
Have a look at how the DS1307 stores time.
"23:15" for example would require 5 bytes.
The DS1307 uses just two, namley $23 and $15 in BCD format.

 

[CliveWilcox]

BB

You know, as soon as i pressed the submit button and went to a late lunch i knew it wrong.

Looking at your number example that explains a problem i had with another project - i never did solve it, just put it down to 'ghosts in the machine'. This is only my third proper project and i tend to be slow a stripping away the outer layers and breaking it down into little chunks.

You are as normal correct on the display after storage option. After all, once it has been displayed its gone. I read over lunch the storage of the temp data - so that makes a lot of sense to store as BCD. As to special formats - ooooh, KISS approach for me. I may well have to explain it to someone, and if i go to sleep i will forget it.

I am going to use one wire if possible, everywhere if possible. If i can find a I2C display other than the one i have that i can't get to work so much the better. I have got the I2C temp and humidity devices today. I have not decide on the memory yet.

Time slots, i was looking at about 10secs or may five secs, so to have 700ms floating around is not a problem and i don't think the 1sec delay will cause a headache either. Its not a high precision data collection requirement. I tend to like things that are about (ish). It makes for a simple life.

 

[BeanieBots]

Temperature stored as BCD. Hmm... not a very good idea.
The raw data is binary, best to keep it that way.
By all means do the timestamp as BCD, that would make sense.

By the way, the DS1307 has 57 bytes of battery backed RAM.
I find it particularly useful for storing alarms or other time related data.
It means you don't need to issue different I2C slave commands between getting time data and alarm data. Could be an option for your "start of log" timestamp.

Maybe now is a good time summarise what you have so far so that you can start finalising the design and know how much EEPROM you need or how many datapoints you can record in an easy design. (KISS).

Now that you how & why these questions are important, maybe re-cap with your sensors and what is the RAW data format?
eg For temperature, will you be using 8-bit or 12-bit?
For the analogue sensors, will they be 8-bit or 10-bit?
Record EVERY sensor EVERY 10 seconds?
(how quick does temperature &/or humidity change?)

Armed with that data, the rest is easy to work out

 

[BCJKiwi]

If you check the sample code referenced in my previous post, you will see the data is stored 'raw'.
When exported to the uAlfat, it is formatted to a .csv format. The advantage of the uAlfat is that it is a FAT device (micro SD card in this case) so is just plugged into the PC and the file opened in Excel or whatever you want.

Re the clock SQW - its just a 'heartbeat' square wave pulse of 50% duration at the rate set - typically 1 sec - i.e. 0.5 sec on /0.5 sec off. So count this up to the 10 sec - code loop time. Is that what you meant by 'goes in steps'?

 

[boriz]

Also...

A 12-bit reading only needs 12 of the 16 bits in two bytes, so there are four spare (unused).

So for every 3 stored readings (3*16 bits), there is sufficient spare space for a fourth 12-bit reading (3*4 bits). So you could have four readings stored in 6 bytes, rather than 4 readings stored in 8 bytes. Saving 25% of your memory.

Also...

Do you really need to store all 12 bits? What is the maximum and minimum possible temperature reading? If you only store the reading MINUS the minimum, then add the minimum later when analysing the data, you can save even more space. You might even be able to get it down to one byte per reading without losing any accuracy

 

[boriz]

EG:

Let’s say that your expected range is 0c – 50c. And you need a resolution of 0.5c. That’s 100 unit’s (100*0.5). If you need 0.25c resolution, then you have 200 units. 0.2c = 250 unit’s. Ideal for 8 bits

 

[boriz]

So you don’t store the absolute temperature. You store the number of 0.2 degrees c units ABOVE your minimum.