Reading from external storage, but which?

J

Jeroen

New Member
Hi,

When designing a new solution I've ran into the following challenge:
for a given application I need to find values in a storage table based on date and time. Including the key(date, time)it adds up to 615kB total of data storage that contain values I might need. I could leave out the key and sort it properly so I can calculate the offset in the data where I want to be and that might reduce the data by 50%. This data is foreseen to be refreshed once a year. It originates from a webservice, from which a stream is captured, processed and cleaned into a raw datafile on the pc that can be processed by ie. pc-applications. But I want a picaxe using this data in a standalone configuration. I could save some more data by recalculating date formats (i.e. to julian).

I do have an eeprom burner but it supports only 28Cxxx series maximum so no i2ceeproms. I have limited capabilities in electronics engineering. I have built picaxe applications that used i2c but have no experience in bitbanging in order to get i2c devices to work (tried once, could not get it to work). The whole thing should be running standalone, there are however no power issues.

What type of storage can you suggest me?
 
hippy

hippy

Technical Support
Staff member
The Vdrive allows a USB memory stick to be connected to a PICAXE through a serial or SPI interface, and there are similar products for SD Card and probably other types of memory cards and sticks.

There are large memory chips which can be connected directly to a PICAXE using SPI or I2C but then you need a means of getting the data into them. Some of the larger PICAXE's shuld be able to interface directly to 28Cxxx Eprom but may need additional hardware.

Your biggest application problem is going to be speed of access and finding data. You may need to sort the data on the PC, perhaps even supply an additional index file to aid in quickly locating the data.

Before deciding on memory and solution it's really necessary to know what you need to achieve and the parameters that's done in; response time particularly, plus budget available. If you are working your way through the data sequentially there may be far better solutions than if you have to search through it. An overview of what your application is always helps people get a better feel for what you're doing and what may be the best approach.
 
T

Texy

Senior Member
I would still go with i2c eeprom option.
You could use an 18x and write a (seperate) basic program to write the data into the eeprom.
I assume the data is available to sort and condition in excel ?
I have just, this afternoon, programmed a 24lc256 with my altimeter lookup table (1024 words) using an 18X picaxe. I had to manually program 256 bytes at a time because I used the internal eeprom data storage area to temporarily hold the data in the program. I just copied and pasted the data
into the basic program, offset the address range by 256 bytes at a time, run the program to program the eeprom, increased the address range by 256 ,copied the next 256 bytes of data from excel, ran the program again......until I had copied all the data from the lookup table. I,m sure it could of been done a lot better than that, but it did the job. I know it is tedious, but for me its a one time job. For you, only once a year.
Note that you do not need to bit-bang i2c with an 18X, or other 'high-end' picaxes.
Hope this helps,
Texy
 
J

Jeroen

New Member
Goal of the project is to create a device that represents the actual sea level on a given place on land. There are two ways: a mechanical: a stepper that "hauls" a representation of the sea level at correct altitude, or a led bar that shows the actual height by lighting leds using a cascaded led bar driver. I might build both options, just to see the difference in sensation. The stepper option causes a lot of state problems in case of CPU restarts that I haven't thought about how to solve yet.
Usually tidal calculations are done using 5 functions of time (and date). However, since it should be representing a given part of the Northsea, 94 (!) functions should be used, corrected with historical data. This will not work for the picaxe, and I even quit building a pc app when I found the data available as a service. So I generated "yearfiles" that contain correct tidal height for every 10 minutes in a given year. The yearfiles are relatively big therefore. I want to read the correct value from that file. The actual date/time should come from a DS1307. As the data is stored sequentially I should be able to calculate the correct addresspace on the medium where the data resides, but it depends on the storage whether that is possible or necessary.
Since the granularity of my data is only 10 minutes, I have somewhat less than 10 minutes to find the next applicable value in the data. This means that I have no real performance issues accessing the data (I hope).
I do not mind to reload the data once a year, but it should be a very easy job, to be done using not much more like a laptop, so preferably without using i2c interfaces etc.

Thanks for the Vdrive hint! That looks promising, especially since the data is immediately interchangeable with any pc program. Best solution that takes care of the data maintenance. However, I could not find any coding examples reading values from files stored on a stick, only dataloggers that are writing?!? Has this already been done?
 
B

BCJKiwi

Senior Member
Sorry to knock VDrive but I had no joy using a 28X1 with SPI and this device, despite many posts and attempted assistance.

Could get Serial to work with a lot of code but the project did not have hserial availble as it was needed for something else.
Also VDrive does not support i2c

Solution was to use uAlfat - similar price, much easier to use and supports i2c (and serial and SPI).
Used the uAlfat-TF unit (TF=TransFlash=MicroSD). These are available through resellers or direct from GHIElectronics.


Addendum
Just re-read the need to read from the file.
With the uAlfat this is a 3 step process;
1. open file for read
2. seek to position in the file.
3. read number of bytes (starting at that position)

The file will remain open if not closed.
The file pointer will remain at the end of the last data read

Would recommend the 4 step sequence for safety
1. open
2. position
3. read
4. close
 
Last edited:
hippy

hippy

Technical Support
Staff member
One 8-bit sea level reading every 10 minutes is 6 bytes per hour, 144 per day, 52560 bytes per year. You should be able to convert your raw data into something which will fit a 64KB I2C Eeprom, maybe even smaller with some encoding ( store +/- changes rather than absolute values ).

That would be the cheapest and easiest option and probably most reliable. Very easy to access and you'd be able to near instantly jump to any time of any day if needed ( after mains power fail for example ) and you could interpolate values between any two 10 minute samples.

It would be possible to write a PC-based utility program which will download data to the Eeprom via the PICAXE. Include the download handling in your PICAXE and you can re-program data without removing the PICAXE from the exhibit / display model.

It may sound like a lot of work but should all be fairly straight forward.
 
M

MearCat

Member
BCJKiwi said:
Sorry to knock VDrive but I had no joy using a 28X1 with SPI and this device, despite many posts and attempted assistance.

Solution was to use uAlfat - similar price, much easier to use and supports i2c (and serial and SPI).
Used the uAlfat-TF unit (TF=TransFlash=MicroSD). These are available through resellers or direct from GHIElectronics.
Click to expand...
Have you used the uAlfat-USB?? ANy chance it would work better then the Vdrive2?
 
B

BCJKiwi

Senior Member
As advised the uAlfat used was the uAlfat-TF (Micro-SD). I was not picky on the device type and went for the smallest and cheapest.

USB interface unit is bulkier and more expensive as it has an additional chipset to do the USB part.
As far as the instruction set and capabilities are concerned they are almost identical - there are a couple of different instructions available as USB offers some different features (such as device detection).

As for working better, if you are going to communicate with serial then VDrive2 (in my experience) will work but my preference is i2c which is not supported on VDrive2.

Don't know where you are, but if near NZ and you decide on the VDrive2 I have one sitting here unused, PM me if interested.
 
Last edited:
M

MearCat

Member
I also have a VDrive and have also found that it does not support i2c. I've had a few other little issues with file handling I'm not happy with, so was looking out for an alternative option. I might purchase the uAlfat-USB and do some testing...

Thanks!
 
You must log in or register to reply here.
Top