I think, people who have just skimmed through the text may get the wrong idea Apatar and Talend, though they are sometimes compared are not designed for the same purpose.
Apatar is not made to deal with large amounts of data. The “zero code” choice limits the flexibility of the software. Anyway, Apatar is, indeed, more geared towards data mashups. Performing ETL requires a tool that can keep up with the amount of data that may be encountered in the reality, and in the open source world, I think Talend is a good choice.

Ichiro.

Slowly as a slowly changing dimension may be, if you take the typical case of a customer dimension, you can often see that the same record gets changed multiple times a day or a week and then not anymore for months on end. More often than not, some typo was detected and data re-entered. As such, you get multiple changes for the same record in your input stream. If you send this same record to multiple copies of a step, you can not just get cache-misses, but also create deadlocks if the threads use multiple database connections.

However, since we allow data to be partitioned in PDI we can as such launch a “dimension lookup/update” in parallel guaranteeing that data is NOT in the cache of another step. That is because we can send the same natural key to the same copy of the step each time it passes.

We can extend that principle to multiple servers running multiple copies of the same dimension updater as well.

HTH,
Matt

Andrew

I bet if you look at the generated code Talend will do a select * type operation on the target table and store all this data in a hash.
The problem with this approach is that it isn’t parallel processing safe (what if a separate workflow updates values in the dimension while it is being used)
And it’s also NOT scaleable, if I have say a customer dimension with millions of customers I will quickly reach the RAM limits and the transformation will then become very very slow.
You may be able to do a join with the source database and restrict the values you cache in the lookup to those in your source but this is going to affect your performance and require you stage everything.

All sarcasm aside, our SCD implementation has been around for about 4 years now, so I’m not surprised it’s running well either. However, I resent that you are trying to imply I’m trying to “fix” our numbers somehow.

All our transformations and input files we use for testing are completely out in the open and if you want, you can run them yourself. I even included links to the test used in question for your convenience. (v2.5 vs 3.0 so you have both)

That is a lot more than I can say about your test claiming our SCD implementation is incredibly slow. So you probably forgot to put an index slowing our transformation down or something. Who knows, right? A local or a remote database? What type of database? Index configurations? A lot of things can influence the numbers. You mention creating the SCD, what about lookups, partial updates with mostly inserts, mostly updates, etc?

If you are interested in those numbers for PDI, I put the link to the transformations and the unit test-cases in my previous post.

By the way, it should be fairly simple to do a Type 3 SCD in PDI. You can look up the original or (more often required) previous entry of a particular column with a second lookup step before you do the actual update.

HTH,
Matt

For a typical result (v2 versus v3), see for example an older run here: http://kettle.pentaho.org/svn/Kettle/trunk/test/org/pentaho/di/run/RunResults-Matt-20070522.txt

I quote for v3: 250,000 row at 12,529 rows/s = 20 seconds. (Initial load)

I still would like to know how you can have a Slowly Changing Dimension without a surrogate key

Matt