Jag,
#1. What kind of database do you plan on using?
#2. What kind of API do you plan on using to connect to the database?
#3. I've done this before. If you bulk load the data into a permanent table and then use a UPDATE x SET y = (SELECT z from aa where aa.bb = x.cc) in Oracle, or UPDATE x SET y = aa.z FROM aa WHERE x.cc = aa.bb in SQL Server
you would be able to pull this off with a single processor very quickly.
With SQL Server, it would not be hard to do. Same with Oracle or Postgres. You're not doing this kind of work with mySQL (good for inserts and deletes, but not updates from what I know, this is an involved subselect query).
PM me if you need more info
.
Mitch