Comments on: Hadoop World: Sqoop – Database Import for Hadoop http://www.cloudera.com/blog/2009/12/hadoop-world-sqoop-database-import-for-hadoop/ Hadoop and Cloudera's Products and Services Fri, 10 Feb 2012 20:11:24 +0000 hourly 1 http://wordpress.org/?v=3.3.1 By: Arvind Prabhakar http://www.cloudera.com/blog/2009/12/hadoop-world-sqoop-database-import-for-hadoop/comment-page-1/#comment-16674 Arvind Prabhakar Fri, 18 Nov 2011 16:54:53 +0000 http://www.cloudera.com/blog/?p=1694#comment-16674 Hi Sathi, Sqoop does allow you to do incremental imports based on time-stamp columns or numeric columns. During such imports, Sqoop populates the data files in the same target directory using names that start from where the previous job may have left off. That way the target directory is always current with the latest changes. Regarding the storage of a transactional table that is changing everyday, one way to use it for ETL use-cases would be to load it up in HBase. Storing it in flat files may not be a good idea since that could lead to duplication if previously imported data is updated. Thanks, Arvind Prabhakar Hi Sathi,

Sqoop does allow you to do incremental imports based on time-stamp columns or numeric columns. During such imports, Sqoop populates the data files in the same target directory using names that start from where the previous job may have left off. That way the target directory is always current with the latest changes.

Regarding the storage of a transactional table that is changing everyday, one way to use it for ETL use-cases would be to load it up in HBase. Storing it in flat files may not be a good idea since that could lead to duplication if previously imported data is updated.

Thanks,
Arvind Prabhakar

]]> By: sathi http://www.cloudera.com/blog/2009/12/hadoop-world-sqoop-database-import-for-hadoop/comment-page-1/#comment-16655 sathi Wed, 16 Nov 2011 07:28:16 +0000 http://www.cloudera.com/blog/?p=1694#comment-16655 Does the current version of sqoop have a way to capture only delta?if I have initial load of table A under /original ,and then I import incremental under /delta ,is there a better way to store a combined data under /latest which is basically merge of /original and /delta In a standard ETL pipeline,If I am storing a transaction table which is changing everyday,what is the best way to store this evolving data within hdfs? Does the current version of sqoop have a way to capture only delta?if I have initial load of table A under /original ,and then I import incremental under /delta ,is there a better way to store a combined data under /latest which is basically merge of /original and /delta
In a standard ETL pipeline,If I am storing a transaction table which is changing everyday,what is the best way to store this evolving data within hdfs?

]]> By: Jeff Hammerbacher http://www.cloudera.com/blog/2009/12/hadoop-world-sqoop-database-import-for-hadoop/comment-page-1/#comment-13121 Jeff Hammerbacher Wed, 11 Aug 2010 17:15:29 +0000 http://www.cloudera.com/blog/?p=1694#comment-13121 Hey Arushi, Unsqoop is now available in CDH3b2 (see http://www.cloudera.com/blog/2010/07/what’s-new-in-cdh3b2-sqoop for more details), so go and get it! Additionally, we have added a Sqoop task to Oozie (http://www.cloudera.com/blog/2010/07/whats-new-in-cdh3-b2-oozie), so you can tie the import, processing, and export phases together in a single Oozie workflow. For further questions on Sqoop, check out the Sqoop mailing list at https://groups.google.com/a/cloudera.org/group/sqoop-user. Regards, Jeff Hey Arushi,

Unsqoop is now available in CDH3b2 (see http://www.cloudera.com/blog/2010/07/what’s-new-in-cdh3b2-sqoop for more details), so go and get it! Additionally, we have added a Sqoop task to Oozie (http://www.cloudera.com/blog/2010/07/whats-new-in-cdh3-b2-oozie), so you can tie the import, processing, and export phases together in a single Oozie workflow.

For further questions on Sqoop, check out the Sqoop mailing list at https://groups.google.com/a/cloudera.org/group/sqoop-user.

Regards,
Jeff

]]> By: Arushi http://www.cloudera.com/blog/2009/12/hadoop-world-sqoop-database-import-for-hadoop/comment-page-1/#comment-13118 Arushi Wed, 11 Aug 2010 09:38:39 +0000 http://www.cloudera.com/blog/?p=1694#comment-13118 Any updates on unsqoop? I wanted to get data using hadoop, modify it using mapreduce and put it back into the mysql. Is it possible with latest sqoop or shall we wait for unsqoop to send data back to MySql? Any updates on unsqoop?

I wanted to get data using hadoop, modify it using mapreduce and put it back into the mysql.

Is it possible with latest sqoop or shall we wait for unsqoop to send data back to MySql?

]]> By: Oded http://www.cloudera.com/blog/2009/12/hadoop-world-sqoop-database-import-for-hadoop/comment-page-1/#comment-8191 Oded Mon, 14 Dec 2009 10:17:40 +0000 http://www.cloudera.com/blog/?p=1694#comment-8191 any updates re unsqoop? any updates re unsqoop?

]]> By: Daily Digest for December 11th http://www.cloudera.com/blog/2009/12/hadoop-world-sqoop-database-import-for-hadoop/comment-page-1/#comment-8100 Daily Digest for December 11th Fri, 11 Dec 2009 11:43:34 +0000 http://www.cloudera.com/blog/?p=1694#comment-8100 [...] Shared Hadoop World: Sqoop – Database Import for Hadoop. [...] [...] Shared Hadoop World: Sqoop – Database Import for Hadoop. [...]

]]> By: Chetan Conikee http://www.cloudera.com/blog/2009/12/hadoop-world-sqoop-database-import-for-hadoop/comment-page-1/#comment-8084 Chetan Conikee Fri, 11 Dec 2009 00:08:12 +0000 http://www.cloudera.com/blog/?p=1694#comment-8084 Typically Netezza and TeraData are bundled with their own proprietary OBDC bindings [and] ship with a JAR file which essentially is a ODBC-JDBC bridge.. So if you have the specified bridge jar you should be able to Sqoop from/to Netezza or TeraData Typically Netezza and TeraData are bundled with their own proprietary OBDC bindings [and] ship with a JAR file which essentially is a ODBC-JDBC bridge..
So if you have the specified bridge jar you should be able to Sqoop from/to Netezza or TeraData

]]> By: aaron http://www.cloudera.com/blog/2009/12/hadoop-world-sqoop-database-import-for-hadoop/comment-page-1/#comment-8075 aaron Thu, 10 Dec 2009 19:53:20 +0000 http://www.cloudera.com/blog/?p=1694#comment-8075 I'm not certain. Sqoop uses JDBC and generates standard SQL, so it can connect to many database vendors out-of-the-box. Some vendors require nonstandard SQL, or implement JDBC drivers slightly different than the norm, for which we have to write special-case code. We haven't explicitly tested it with Teradata. I'd love to hear feedback as to how that experiment works out. I’m not certain. Sqoop uses JDBC and generates standard SQL, so it can connect to many database vendors out-of-the-box. Some vendors require nonstandard SQL, or implement JDBC drivers slightly different than the norm, for which we have to write special-case code.

We haven’t explicitly tested it with Teradata. I’d love to hear feedback as to how that experiment works out.

]]> By: Jing http://www.cloudera.com/blog/2009/12/hadoop-world-sqoop-database-import-for-hadoop/comment-page-1/#comment-8073 Jing Thu, 10 Dec 2009 19:39:59 +0000 http://www.cloudera.com/blog/?p=1694#comment-8073 Does Sqoop work with Teradata? Does Sqoop work with Teradata?

]]>